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ABSTRACT 



This Starter Kit is designed to help organizations and 
individu<"3ls who wish to offer human-mediated information services via the 
Internet to users in the K-12 community. A six- step process is proposed for 
organizations to follow in creating an "AskA" service. This process addresses 
all aspects involved in building and maintaining an AskA service from 
preliminary research of the digital reference field to ongoing evaluation 
techniques . Each module of the Starter Kit describes one of the six steps and 
includes statements of goals and objectives; information through explanation 
and practical examples (case studies from existing digital reference services 
like AskERIC, the National Museum of American Art Reference Desk, NASA's Ask 
the Space Scientist, KidsConnect, and Ask Dr. Math) ; and opportunities for 
interaction (worksheets) . The six core modules are: informing; planning; 
training; prototyping; contributing; and evaluating. Although listed in a 
linear fashion, the six steps can be completed out of order. Module summaries 
are included at the end of each chapter. A concluding chapter offers some 
final notes and a glimpse into the future of digital reference. Contains a 
glossary, AskA service directory, and AskA service blueprints. (AEF) 
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Introduction 

Module Profile 

This module is intended to welcome readers to The AskA Sfarfer Kin How (o Build aiul 
Maintain Digital Reference Services (rel'crred to as the Starter Kii) and provide a road 
map to the resource. At the conclusion ol‘ this module, you will be able to achieve the 
following goal: 

Goal Use ihc St iricr Kit as a guide in building and maintaining a 

new digiiai reference, or AskA, service. 

Prerequisites 1'he Starter Kit is intended for organisations interested in olTcring 
digital reference service to users in the K-12 community. Such 
organizations may include libraries, professional associations, 
government agencies, academic institutions, and companies that 
specialize in a given subject area or process. This resource is directed 
towards individuals planning to serve as .AskA service administrators 
or coordinators but could be used as a reference for other staff as well. 

OI)jeclives At the completion id' this module, you will be able to achieve the 

I’ollowing objectives: 

1 . Understand the purpo.se and organizational structure of the 
Starter Kit. 

2. State the six steps in building a digital reference serv'ice on 
which the Starter Kit is based. 

3. Understand the goals of the \drtual Reference Desk Project and 
the benefits of building an ^\skA Serv ice. 
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What Is the Starter Kit? 



I'hc AskA S (drier Kif: How to Build and Maimciifi Di^^ifctl Reference Services (also 
rcrciTcd to as the Starter Kit) is designed to help organizations build and maintain digital 
reference serviees (referred to throughout this doeument as AskA services). This Starter 
Kit is intended for organizations that wish to offer human-mediated information services 
via the Internet to users in the K~12 eominunity. Such organizations include associations, 
libraries, governmen agencies, and companies that specialize in a given subject area or 
skill. While this resource is geared towards service to the K-12 community, the structure 
and information have wider application in the library and business communities. The 
information and suggestions made in this document are based on research by the Virtual 
Reference Desk project (see “What is the Virtual Reference Desk Project?") and input 
from a panel of representatives from exemplary digital reference services. 

The Starter Kit is intended as a set of sell -instructional modules for AskA service 
administrators and other staff. Modules 1 through 6 prc.sent the six steps involved in 
building and maintaining AskA services (see “Starter Kit Structure" below) as self- 
insiructional, interactive modules. Following each module, the reader will achieve a set 
of goals and objectives as outlined in each module pi'oHle. Each module consists of 
infoi'mation, c.xamples, and in many cases intei'active work sheets for the reader to recoi'd 
important information. Organizations may read the whole Starter Kit from stait to finish 
or coneenti'atc on individual modules as they apply to individual service issues. 

Starter Kit Structure 

I’hc Stai'ter Kit proposes a six-step process for organizations to follow in ci'cating an 
AskA service. This process addresses all aspects invob cd in building and maintaining an 
AskA ser\’ice from preliminary research of the digital reference field to ongoing 
evaluation techniques. Each module of the Starter Kit describes one of the six steps of the 
overall process and includes statements of goals and objectives for the reader to 
accomplish; information through explanation and practical examples (e.g., case studies); 
and opportunities for interaction (e.g., work sheets). The six core modules arc outlined 
below’: 

1. Intbrniing: Collect information on the general digital reference field and existing 
AskA services. 

2. Planning: Determine the best way to build and maintain digital reference service 
within a given organization. 

3. Training: Plan, produce, implement, and manage training programs for service staff 
and information specialists. 
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AskA Starler Kit Introduction 



4. Prototyping: Crcaie a prototype and conduct a pilot test of an AskA service. 

5. Contributing: Manage service development and operations and build partnerships to 
gain necessary support and share processes. 

6. Evaluating: Plan and implement evaluation of AskA service and use results to 
improve service. 

Although listed in a linear fashion, the six steps can be completed out of order depending 
on the needs and experiences of an organization and its AskA service. See Figure 0- 1 for 
a graphical representation of the Starter Kit structure, including goals and objectives of 
each module. 

Module summaries arc included at the end of each chapter to highlight key issues for 
quick reference and review. Wrapping up the Starter Kit is a conclusion, which offers 
some final notes and a glimpse into the future of digital reference. Other features, 
including a glossary, case studies, and interactive work sheets, arc included for further 
explanation, illustration, and application of ideas. 



ri^un* 0-1 Six Slups to Uiiildin^ and Maintaining Digital liel'ci'cncc Services 
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NN'hat Is the Companion Research? 

Much of the Starter Kii is based on a year-long study conducted by the Virtual Rej'crcnce 
Desk project. This in-depth qualitative study, Buildii ond Mainfednin^ hitenict 
Infonualion Services: K-12 Di^ijed Reference Services tLankes, 1998), is available from 
the ERIC Clearinghouse on Information Si Technology. Many of the pertinent pieces 
have been either reprodiiecd or simplified for this work. Hou ever, the stud)’ goes into 
much more detail on the conceptual foundations of this work and the methods used to 
create the blueprints presented in Module 2. This study is not essential for the 
understanding or use of the Starter Kit, but it provides additional detail and supj)ort for 
the scholarly aspects of digital reference work. The Starter Kit complements the research 
study as a practical guide to creating digital referenee services based on the descriptions 
produced by the study. 



What Is the V irtual Reference Desk Project? 



The Virtual Reference Desk project is dedicated to creating the foundations for a national 
cooperative digital icfercnce service. The project is sponsored by the National Library of 
Education (NLE) and the ERIC Clearinghouse on Information Sc Technology with 
support from the White Mouse Office of Science and I'cehnology Policy. 

The Virtual Reference Desk seeks to identify and provide the resources necessary to link 
all K-12 community members (c.g., students, educators, j^arents) to necessary expertise 
in order to satisfy information needs. The goals of the project include research into 
cuirenl ways in which K-12 community members receive answers to questions on the 
Internet and the development of a national collaborative network of Internet-based 
question answering services. The Virtual Reference Desk currently provides a Web-based 
system to help the education community locate existing AskA services and infoimational 
and instructional resources to help organizations build new AskA services. 

The initial development of the Virtual Reference Desk project included the formation of 
an expert panel of representatives from existing digital reference services. In sharing their 
experiences in organizing and managing AskA services, the panel members provided 
important guidance in creating the vStarter Kit. 




AskA Starter Kit 



Introduction 



Module Summary 

This seciion provided an introduction to the AskA Snincr Kii and the \hriual Reference Desk project. 

• The purpose of the Starter Kit is to guide organizations in building and maintaining digital reference, 
or AskA, services. 

• The Starter Kit is tu'ganized as a set of self-instructional modules based on a six-step process \Ar 
creating an AskA service: 

1. Informing 

2. Planning 

3. Training 

4. Prototyping 

5. Contributing 

(). Hvaluating 

• The \’irtual Reference Desk project, sponsored by the National Library of Ediieaiion and the ERIC 
Clearinghouse on Information & Technology, is researclung current ways in which K-12 community 
members receive answers to questions on the Internet and is developing a national network of digital 
reference services. 




Informing 


Planning 

Training 

Prototyping 

Contributing 

Evaluating 





Informing 



Gathering Intbrinatioii on the 
Current Digital Reference Arena 

Module Profile 

This module is designed lo assist organizations in eondueting preliminary research on the 
digital reterenee field in preparation for creating new AskA services. At the conclusion of 
this module, vou will be able to acliieve the followim: goal: 

w cr 

(Joal Collect information on the general digital reference field and 

e.Msling AskA services. 

Prerequisites Before an organization considers providing digital reference service, it 
must identify its special area of expertise. I'his area will obviously 
reflect the focus of the overall organization or special project (see 
Module 2; Planning). Expertise can lake the form of a particular 
subject area, such as matli, physics, or social studies, or a skill area, 
such as information problem-solving, teaching, or parenting. 



Objectives At the completion of this module, you will be able to achieve the 

following objectives: 

1. Define digital reference, or AskA, services from different 
perspectives, including library, education, and business. 

2- Identify gaps in existing AskA specialization areas. 

3. Identify general issues in operating an AskA service. 

4. Identify quality characlcrisiies of K-1 2 digital reference 
services. 

3, Locate information on existing AskA services using the AskA-i- 
Localor. 

6. Locate information on the potential user population to 
determine initial need fur service. 
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Module 1 



1.1 Introduction to Digital Reference and AskA Services 



What Are I)ij>ilal Reference and AskA Services? 



Digital reference ser\’ices are Inlernci-biised question-and-answer services that connect 
users w ith individuals who pos.^ess specialized suhjcct ov skill expertise. As opposed to 
static Web pages, digital reference serx’ices use the Internet to place people in contacl 
with those individuals who can answer specific questions and instruct users in dcs’cloping 
certain skills. Digital reference scr\’ices are also rderred to as AskA ser\'ices, because ol’ 
such service names as Ask A Volcanologist, Ask A Scientisu etc. 

Hundreds of digital reference services exist today, many of them serving the K~i2 
community. Some examples include 

• AskERIC, a project of the Educational Resources Information Center (ERIC) since 
1992, which places teachers, librarians, parents, and others in contact with 
information specialists who pros’ide pointers to education information sources 
(Lankes, l99o). 

• KidsConnect, a project of the American Association of School Librarians, which 
places K-12 students in contacl with school library media specialists who help them 
find intemet and non-Internet resources to answer their questions (Bennett, 1997). 

• Ask Dr. Math from the Math Forum at Swarthmorc College, a team of 
mathematicians (or “math doctors") who respond to K-12 students' questions about 
math l\up://forw}\,s\\'anl\t}U)rc.cdii/d}\n\aili/ 

• Ask A Scientist from the MAD Scientist Network (of the Medical School at 
Washington University in St. Louis), which allows students to ask questions of 
science experts lutp://\\'\\'\\\}nadsci.or}:^ 



While many AskA services cater to K-12 audiences on topics relating to curriculum 
areas, digital reference services also represent other contexts, including library and 
business. The context of digital reference (in addition to other factors) often dictates the 
overall goals, day-to-day ojicrations. and scope of the service. Digital reference has its 
roots in the library field hut affects and draws from experiences of education and business 
contexts. 
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Informing Module 1 

Digital Referoiice From an Education Perspective 

The Inlcrnct can be an dTeciive and motivational learning tool when used in context with 
educational goals and objectives. It adds an interactive dimension to K~i2 education by 
enabling students, educators, parents, and others to communicate with each other and 
with subject and skill experts all over the world. Students can learn Tirslhand about life in 
Antarctica by corresponding with a incicorologist who regularly visits the continent 

Library media specialists can discuss lesson-planning ideas with 
colleagues on a 7,00()+-mcmbcr e-mail discussion group called LM_NET 
[http://cncir.s\'i\cdit/lnijict/). Parents can receive advice on helping children with study 
skills by contacting educational researchers at Parents AskERIC from the Na‘- ;nal Parent 
Information Aelwork {htip://cricps. cdAiiuc.edu/npin/), 

Digital reference services maximi/.e this inicractix c nature of the Internet by prox iding 
expertise m response to questions as well as direction in enabling users to answer their 
questions independently. AskA scrx'iccs normally respond to individual queries on a one- 
lime basis; this differs from mentoring programs, which conned students and teachers to 
subject-matter experts for longer term cuiTiculum-bascd projects (for example, .see 
Sanchez & Hairis, 1996). 

Howcx’cr, most exemplary digital reference sciwiccs agree that it is not enough for experts 
or information specialists to simply offer an.swcrs. Digital reference serx ices can prepare 
users in the K-12 community — students, teachers, p::rcnis, etc. — to become effective 
information problem solvers. In other words, digital reference services can play a 
significant rule in helping people develop skills for critical thinking and assessment of 
information resources to apply to any context at any stage of life.' 



I'or more inlormalii'^n on how digital ictcrcnee Ncr\ ices can eairy out then in>lrueIioiial role through 
^c^po^se^. see Kaso\\.iiz ( I99H). 
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Inlorming ^ _ _ _ _ Module 1 

Digital Reference from a Library Perspective 

Libraries arc becoming an increasingly visible presence on ihc Internet, with many 
launching Web sites and offering digital pathfinders to other online resources and 
collections. As with many oi'ganizations that have established an Internet presence, 
libraries arc cuirently exploring ways in which to expand their services by interacting 
with users and responding to inquiries via the Internet. For libraries, this type of viraiai 
interaction is an extension of the face-to-face or telephone reference service already 
offered. Digital reference, sometimes refeiTcd to as electronic reference in the library 
field, allows library users to access a human intermediary at their own convenience 
without having to physically enter the library facility. 

Libraries as Askj\ Services 

L.ibrarics, like subjcci-oricntcd AskA services, engage in digital reference by directing 
users to appropriate online or print resources and7or providing factual information. 
Occasionally user queries must be clarified, and an online reference interview may be 
conducted to help define the user’s information needs. Examples of library-based digital 
reference services arc seen mostly in academic and public libraries. 

While many academic libraries have established digital reference services, most restrict 
their services to their local academic communities only. Common methods of limiting 
access to services include password protection or Internet Protocol (IP) address mapping. 
Some academic libraries, such as Roger Williams University Library in Bristol, Rhode 
Island’, cuiTcntly accept questions from the general public, but these libraries tend to have 
modest question loads at the present time. 

Public libraries, though slower to imp!* lent digital reference services than academic 
libraries, typically accept questions fro. ^ the general population. Some libraries such as 
the Seattle Public Library (SPL) restrict their services to valid SPL cardholders only, 
while others do not publicize their services in order to limit the number of questions 
received. 

The following pages discuss digital reference in a library context in terms of the impacts 
of the Internet reference service, changing roles of reference librarians, and digital 
libraries. This discussion is included as an excerpt from the companion research report 
Building ami Maintaining Internet liifonnalun} Sendees: K~12 Digital Reference Sen'ices 
available from ERIC Clearinghouse on Information and Technology (Lankes. 1998). 

Information from personal communication. Su.san McMullen, Inibrmaiion Resource.' Librarian at Roger 
W'illiams University Ubrary November 14, 19^8. 
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Denning Digital Reference Services 

Foi' the purpe^scs (his study, rofeivncc scrs iccs are defined as mediated 
inlei j'aces hclvveen users in an '‘anomalous state of knowledge” (Belkin, 19S0) and a 
eolleclicMi of information (Sutton, 1906, p. 131-3). The user's anomalous stale of 
knowledge, also referred to as a gap in cognitive understanding (I)ervin ^ Nilan, 19S6). 
is operationa!i/ed in this study as a question that needs to be answered. This question 
may be expres,sed as an e-mail request or a query to a system (Taylor. 1968). The 
collection is a set of information in the form of documents. Hies and/or knowledge 
(including human expetlise). In this study, all information was delivered to a user 
electronically via the Internet^ 

Mediation between the user and the query is the central topic of reference 
research. Mediation can he performed either ilirough a human expert (such as a reference 
librarian) or an automated interface (such as an online catalog). The primary purpose of 
the interface is to match the user’s information need to the system's organi/aticui and 
capabilities (Taylor. 1968). The mediator (once again, either automated or human) 
becomes the user's advocate to the system or collection. This view of reference is 
maintained in today's electronic reference environment (Sutton, 1996). This study 
concentrated on Itow organizations, specifically K-12 digital reference services, built 
Internet information systems that fuiniled u.sers’ reference needs. Restating the 
concentration of this study using language from the reference discussion above: how 
organi/.alions built and maintained information services that mediated between a user's 
information need and a collection of information. 

Impacts of the Internet on Reference Services 

The literature shosvs significant impacts on refcrenco services prompted by 
greater access to tlie Internet and Inlcrnel tools. These impacts include new skills needed 
by inlormation specialists and reference librarians (Bobp, Kraizert ^ Richey. 1993). The 
Imcrncl is also expanding traditional library collections and improving location and 
access'^ to reference resources (i.c, ready reference materials and pathfinders througli 
World Wide Web sites, access to catalogs and electronic reference .sources through telnet, 
etc). Most significant to this research, the Internet affords reference services the ability to 
conduct entire reference transactions (from specifying u.sers' needs to delivering 
information from the collection) via the Internet (Still ^ Campbell, 1993). 

A great deal of literature has focused on augmenting traditional reference 
.services with Imcrncl resources and capabilities. This literature ranges from evaluation 
criteria for on-line reference sources (Balas. 1995) to discussions of technology u.scd to 
locate and access Internet resources (examples include Feeney, 1993; Bobp. Katzeri & 
Richey. 1993; (lainor A: Foster, 1993; Arms, 1990; Bransc, 1993; Machovec, 1993). In 
the.se discussions, the interface to (lie u.scr remains the same, but the collection is 
expanded to include Inlcrnel resources. These new resources change the reference 
environment. Mardikian and Kes.selman ( 1995, p. 22-3) presented live “rationales for 
c h a n g i n g r e 1 0 1 'C n c e : ” 



‘This can he done through clieniyserver sysicins like ilie World Wide Web or oilier system.s such as o-rnail. 
Tor a discussion on location and access see liiscnbeig and Bcrkowit/-. 1990. p.7. 
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• Increasing access lu resmirces hcyciul llie lihiary uiciworkcd resources including ihc 
InierneO. 

• Lack of goograpliie const rainls lor users (* users ma\’ no longer need lo come to the 
libiary lo oblain informalion"). 

• 'I‘he need to dilTereniialc services lo dilTerenl popiilalions of users { i.e., inside an 
•M'gani/.aiion and oulside an organi/alion ) in ihe face of shrinking budgels. 

• Increases in complexity of inrormalion resources and the need (or special i/ed 
kn(v,vledge, 

• Nev\' opiic^ns (primarily in staffing) for answering reference questions. 

All of these rationales coneemrale on having librarians redefine iheii roles within a 
traditional, geographically defined library setting. 

Changing Roles of Rdercuce Librarians 

These changes in the reference environmeiu focus on the reference librarian and 
the training of that librarian in response lo the “increasingly automated | library] over the 
past three decades" (He &. Knee. 1995 p.7). He and Knee presented the idea of an 
eleeiixMiie ser\ ices librarian. In regards to reference services, they slated, “It is important 
for elceironic .services librarians m be familiar with traditional as well as electronic 
reference sources. Hy learning traditional sources, they will be able to recognize which 
Internet resoLiives may also be \-al liable" (p. 9). He and Knee called for librarians to 
update liieir skills in response to perceived changes to llic reference environment. 
Librarians' skills must also include an ability [o evaluate networked resources, Meriure 
( 1994) discussed the evaluation of networked resources. He drew upon Van House et al. 

( 1990) to define evaluation as “the process of identifying and eolleeling data about 
specific services or aetivilios. establishing criteria by which their success can he Dssessed, 
and determining both the quality ofilio ser\'ice or activity and the degree to wliich tiic 
service or activity accomplishes staled goals and obJeeii\'cs" (McC'lure 1994. p.592). 

The burden of learning and applying the application and evaluation skills of the 
Inieriiel falls upon the lihrarian. The librarian must master the new Internet tools for his 
or her users. The reference librarian acts as “a bridge whieli has technology at one end 
and the useral the other" {('allalian. 1991). Learning, however, is not limited to just 
applications and technology. It also applies to learning to deal with change. McC'lure cl 
al. staled “library staff. , . must learn from their colleagues ir. Mic computing .servicc.s 
how to become more ct)mforlable with the type and rale of change tlial will accompany 
die networked environment" (McClure. Moen & R\’an, 1994). This notion of change and 
die need for technical proficiency is echoed throughout most of the literature coneerniiig 
reference services and the Internet. The use of complexity in this research was a reaction 
lo (lie fact that digital reference services must also deal with a great amount of change. 

AccompaiiNung the changes in reference librarians' skills are changes in llie 
reference librarians' roles, pariicularl)’ in regards to slalTing. Oberg stales 
“paraprofcssionals can and do perform well at a reference desk, freeing librarians to 
coiicenirale on liigher-Ievel tasks" (from Mardikian Kcsselnian, 1995, p.21 ), Mardikian 
and Kessciman presented a three level staffing model to reflect the changing role of the 
reference librarian (sec Table 2-1 ). 
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'I'iibli* 2-1; Mardikiaii iimi Kossfl mail's StafTin^ (a‘vi*Is (P’rorn fiuUdin^ and Muinniiitinjii fnfcnicf 
Information Scryiccs: A-/2 l)i}>Ual PrfiTvnic .Vm /ivs ) 

Level 1; Miiiinuini Human Interveiilinii 

• Scll'-uuided hi 'tiding lours 

• Aiiioinatcd lclcplK>ne answering miK’hincs 

• lieu or Mgiiage 

• lioUer floor maps 

• Library quick guides 

• Siep-hy-siep guides 

• ('ompuior-assisted inslrueiitwi Idi seU-serMec iiisimetion 

• rompuierized inl'ormation kiosks 

Level 2: Library I nterns/T rained Parapmfessional StalT 

• ( ioneral library orierualion and general bibliographic uiMriielum 

• nirectional inquiries 

• Ready ret'erenee searching 

• Bibliographic veririeaiioM t>n OC’IX'. RUN. and the online catalog 

• Assist with search slraiegs r'onmilaiion 

• 'rechnieal assisianee with machine piohlems 

• Basie inromuitional sen ices ^sith ivrerrals as needed 

Level 3: Librarians, Subjecl Specialisis 

• Individual research consullaiions 

• Speeiali/'cd rorcicnee scrs iecs 

• OlTiee hours in depaiiments 

• Member of a rcseaieh learn with leaching faeuliy 

• Liaison aetiviiies wiili departmeiUs 

• Speeiali/.ed instrucuonal sei wecs 

• Imegraie information literacy into tlie eiirneuliim 

• Rcseaieh and devclopmcm effoiis 

• Mediated online searehing 

• ('reate C'Al pn^grams and cvpcrl sNsieins for users 

• Ongoing evaluation and needs assessment 

iTnm this table llie researcher assumed lhai most iuftirmalion specialists \vt)rking in K-12 
digital reference services w'oiild fall inu^ category tiiree with some in category twev 

Acccinpanying this shift in responsibilities for reference librarians (to higher- 
level tasks) is a call for greater collahoralion with lUher types of professionals. LcN\ is 
(1995) believed the infusion of new tools for location and access into libraries means “a 
signinctml upgrading of skills of most librarians and will mean professionals who are not 
librarians will have to be offered positions along side of. or in place of, librarians." 
McClure, et al. ( 1994, p.67) listed partnering with computing services, faculty, and oihu'r 
“external organizations and companies" as critical success factors in building the virtual 
library. One would expect members of digital reference services to he highly 
knowledgeable in technology and Internet applications. However, as discussed in C'liapter 
4, few services have specific detectors for infrasimeture issues such as hardware and 
wiring, instead relying on their larger organizations for such krunvledgc. Indeed, these 
services have formed strong rclationship.s witli computing centers and technical 
organizations as discu.ssed by McClure et al. ( 1994). 

The literature did, howcNcr. allow the researcher to aniicipate that information 
specialists in K-12 digital reference services would have backgrounds other than library 
and information science. Indeed while the two pretest services of ihi.s study (AskERIC' 
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and the fnlernet Public Library) claimed strong backgrounds in formal library training, 
only o.,e of the six services studied (The National Museum of American Art Reference 
Desk) employed a professional librarian. 

Digital Libraries • 

The Internet is also used to provide better access to a library's collection. The 
Internet is used to organize materials for reference patrons"" (Jensen & Sih, 1995) and 
allow patrons access to reference sources such as OPACs^’ (He & Knee, 1995). This 
reference collection literature includes discussions of standards for information 
interchange (Moen, 1992). The literature seems to present a continuum for reference 
services and access in relation to the Internet. There has been a general belief that 
libraries and reference services are headed “towards a virtual future” (Strong, 1996). 
However, this future has not been widely explored. 

Sutton’s (1996) four-part typology of libraries anticipated the expansion of 
reference collections to include the Internet, as well as the use of the Internet to access an 
individual library's collection. This four part t) pology (see Figure 2-1) created a 
continuum from a paper-based (“traditional”) library to a fully “digital” library without 
walls (Sutton, 1996, p. 129). It consists of: 

• Traditional: “a specific place with a finite collection of tangible information bearing 
primary entities like books and journals , . . Idenoted as] paper” (Sutton, 1996, p. 
131). 

• Automated: a mix of paper and digital reference resources and meta-information that 
“point to non-digital media” (Sutton. 1996, p. 135). 

• Hybrid: typified by the use of both print and digital meta-information sources 
(increasingly digital) and the coexistence of both digital and paper primary resources. 
This type of library allows for the first time remote access to “some subset of the 
library's digital collection or to digital reso' ’es”(Sutton, 1996, p. 136). 

• Digital: . . the library as a logical entity. *s the library without walls — the library 

does not collect tangible information bearing entities hut instead provides mediated, 
geographically unconstrained access to distributed, networked digital information” 
(Sutton, 1996, p. 138). 

From this typology, Internet information systems, specifically digital reference services, 
can be seen as “digital” libraries. Since such services transact all information delivery via 
the Internet, they are fully digital. 

One interesting aspect of a digital library’s reference services is the ability of the 
Internet to hide the process of reference services. Still and Campbell noted: 

. . . one big difference Ibetween traditional reference interactions and using the 
Internet for reference workl was that e-mail has made the internal operations of 
the library invisible to the patron; they are unaware of which department handles 
each request. The patron simply asks the question” ( 1993 p. 16). 



'Pairon is a library term synonymous wnh user or oisiomcr. 

^’An OPAC IS an Onluio Public Access Caialog. ll is a compulor daiabasc ihat allows library patrons access to 
informal ion on a library’s collection, 
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The present study was a direct response to the “black box" effect of the Internet. K-12 
digital reference services are being built and used, but it is impossible to determine more 
than the most rudimentary processes within the actual reference process.''^ Most services 
in their public documentation and description concentrate on what the service does, not 
how it is done. 

Sutton (19^6) slated that in a digital library the primary task of the librarian is to 
provide “context” (Sutton refers to Saffo's [ 1994] concept of context). That i.s to say, the 
collection becomes so large (it could be considered to consist of the entire Internet) that 
patrons no longer desire the full range of information available on a given topic, but the 
“best” information. The librarian's role shifts from advocate to a collection to a filter for 
the user. Since the patron is no longer bound by geography (or technology), the user will 
select services based on how well they create a context useful to that user. So the 
selec'ion of K-12 digital reference services could be seen as a selection of contexts. 



Digital Reference Services Summary 

Reference librarianship and reference services have a long and rich history. It is 
clear from the literature that the Internet has a major impact on how reference services 
are accomplished in the networked world. A continuum appears to have been established 
(see Figure 2-1). The continuum starts at a library with no automation, unaware of the 
Internet, and ends in a library as “logical entity” (Sutton, 1996, p. 131) where reference 
services provide context to a globally distributed, fully digital collection. Much of the 
discussions within this literature centers on the role of the reference librarian. The 
librarian or information specialist of the fully digital library appears to be technically 
literate, but versed In traditional reference resources; cooperative with organizations 
outside of the library; prepared to cope with great change; and able to provide context to 
patrons. These characteristics certainly apply to the services studied in this research, even 
though they do not have formal library training. 



’a Muck hox clTccl is used lo rcl'cr to a s\sicm wlicre onlv the inpui and output arc knnun. 1 inuever, one is unuMe to 
deicrminc the means that transform input into (uiipul. 

\Somc .services make such information explicit. For example, ihc AskFRiC service not onlv includes infnrmalum on 
who IS amswcring a question, but also on ine vvay in which thai question was answered. 
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Focus on the Container 
Location bound 










Focus is on Information 
Location unbound 






Kij»ure 2-1 Continuum ot' Libraries: Sutton t LWC) csiahlishos u continuum ot'libraiics from "tuditional” 
that is lutLind by its physical raulilios and conccmralcs on “objects" such as hooks to “digital" in winch the 
network and electronic documents make objects and location irrelc\ant il'roin HuiUlijiy and A/r//nf(///////y 
Inicnicl information Services: K-12 Digital Rclorcnco Sm u cs.) 



This liicrauire informed the researcher as to the importance of this study (to 
coiinlcracl the black box effect created by Internet digital reference services) and built a 
set of anticipated characlciistics of information professionals within these digital 
reference services. This literature led the researcher to expect information specialists 
would be technically literate and from a wide variety of backgrounds. The e.xpertisc on 
the part of digital reference service workers was accounted for in the methodology. 
However, the researcher could also expect tlie builders and maintainers of these services 
to be able to understand and explain both the processes of their Internet information 
services and the technology resources used. This combination (process knowledge and 
teclinical knowledge) supported the choice of K-12 digital rererence services as the 
population for this study. The following sections build a conceptual framework used to 
explore the "digital library" environment facilitated by the Internet. 
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Digital Reference From a Business Perspective 

Business has long understood the in'iporiancc of a human-io-human connection with 
users, whether the users be business partners, customers, or employees. Recent articles 
have highlighted the impoilancc of call centers and help desks in companies (sec 
Bibliography — Business). 

Help desks normally utilize special software to track internal software or product 
problems. These systems assign “ticket number'’ or “incident" reports to staff problems, 
catalog system personnel and their specialties, and store and track answers as they arc 
created. In these systems, a knowledge base is created over the life of the system 
consisling of problems cncounicrcd o\’cr lime. The staff at the help desk lend to be 
specialists or experts. The goal of the system is to track problems and make sure they arc 
answered in a timely manner, while capturing novel solutions for future reuse. 

Call centers operate differently from help desks. These systems lend to be telephone- 
based and suppoii known answers. In these systems a set of known answers exist in a 
knowledge base, and a series of scripts leads a relatively inexperienced customer service 
representative through a predetermined set of steps to determine a set course of action. 
The classic c.xample is the “broken toaster" scenario in which a customer with a broken 
toaster calls a service line. The call is routed to the next available representative. Upon 
making a connection with the cusioiner, a representative launches into a series of 
.scripted question‘d meant to narrow possible known solutions (e.g., “Is it plugged in?" 
“Do the hcatina elements uirn red?" “Do vou smell smoke?"). These systems work in 
areas with a finite set of answers, where the process is to get to a known solution. If a 
known solution docs not exist, the customer is normally taken out of this sx'slcm and 
transferred to another system, such as an engineer, or the item is returned so that another 
department can take a look at it. 

Both of these types of solutions belong to a larger class of software known as work-flow 
solutions. These solutions invoKc a large analysis component followed by a stratification 
of a business process into a series of layers. These layers classically consist of 

• process logic: the business rules and established procedures for acting upon 
information 

• information resources: data, hardware, and software applicalitms that manage and 
manipulate data 

• human resources: experts, managers, and other organizational employees (Stark. 

1 997 ). 
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These layers are put in place so that an organization can effectively process information 
or an event (say a request for information) effectively. By stratifying these layers, 
components at each layer can be replaced without disrupting the' other layer. 

Work-flow solutions arc embracing the Web and Internet. While there have been some 
delays and turbulence as they grapple with a new business environment (for example, 
changing from a costing based on “number of customer service scats” to simultaneous 
connections), these softw'are vendors arc beginning to use the Web to provide common 
interfaces. However, there has been little attempt to create methods of exchange among 
these packages. 

Summary of Digital Reference From Multiple Perspectives 

Building from experiences of libraries, education, and business, organizations have 
already begun to build and operate AskA services. Together, these AskA services 
represent a new, distinct field with common issues, characteristics, and quality standards. 

L2 Current Trends and Issues in AskA Services 

The digital reference field is diverse and dynamic. Existing services range from one- or 
two-person “Mom and Pop" operations to services staffed by over 400 “expert" 
employees or volunteers. Numbers of questions answered range from two to over 200 per 
day (see AskA Capacity Matrix l\ttp:/Av\v\v.vrd.or^/AskA/capacity.htni), Despite the 
dilTerences among AskA services in size, scope, etc., AskA services do share some 
common issues. This section describes the state of AskA services today in terms of 
expertise areas as well as common issues regarding operations and service quality. This 
information is provided to help organizations prepare to start planning their own services. 
It attempts to answer these questions: What’s out there? What should new services 
expect? 

Survey of AskA Services’ Specialty Areas 

AskA services provide information in a variety of subject and skill specialties (e.g., 
health, science, math, etc.). As in any type of business or service industry, it is helpful to 
engage in some market research before providing a new AskA service. An organization 
should review the arena of existing services before offering a new service in a certain 
specialization area. Depending on cuirent representation of a particular topic, an 
organization may choose to focus specifically on a subspecialty (e.g., geography of 
Canada) or to broaden the scope of the service (e.g., geography). 
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With hundreds of new Web sites appearing daily, the Internet can be a rich source of 
information for students, parents, and educators. Increasing numbers of universities, 
organizations, and other entities are offering AskA services to the K-12 community. 
Ho\vc\'cr, despite the proliferation of AskA services, many subject areas remain 
inadequately represented with few or no services in effect. For example, while users 
seeking information from scientists or reference librarians have several services from 
which to choose, those seeking interaction with art and ait education cxpcits find very 
few services available (e.g., see "A Report of Art and Art Education AskA Services" at 
the end of this module). Other subject areas such as history, literature, foreign language, 
and world geography are similarly underrepresented. To fuilher enhance and support K- 
12 teaching and leaniing efforts, all subject areas should be represented for 
comprehensive coverage. 

At the present time, for example, there are no AskA services answering questions on the 
American Civil War, or European or Asian art for the general public. The addition of 
such services, and their personal replies to student inquiries, can capture students' interest 
and further motivate them to explore such subject areas more fully on their own. Due to 
the lack of digital reference services in subject areas such as art, history, and others, 
however, the information needs of many users may be left unmet. Moreover, a potential 
burden may be placed on the few services that do exist in underrepresented subject areas, 
as the question volume may prove to be overwhelming for too few services to effective!)- 
manage. 

The AskA Overviews located at the end of this module provide surve)'s of AskA services 
in two distinct subject areas: art and paleontology. They offer summanes of existing 
AskA services in specific subject areas in order to help organizations plan for the 
development of similar yet distinct AskA services. 

Issues in AskA Services 

Although digital reference services can vary in many aspects including number of staff, 
number of questions answered, technology used, and subject areas covered, they do 
e.xperience many common issues. All AskA services grapple with issues of policy, 
operation, and management. For instance, many services are familiar with the experience 
of starling what they had planned to be a small, controlled question-answer service for a 
specific population (possibly as an outgrowth of another Web resource or initiative) only 
to very quickly become overwhelmed by hundreds of questions from the general Internet 
public. In addition, many services struggle with legal issues regarding liability for 
information provided and confidentiality of user information posted on services' Web 
sites. Common issues arc summarized on the following pages. 
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Lci^al and Ethical Issues 

User Contldentiality. AskA services lhal plan lo make user correspondence public 
through a question-answer archive, etc., should consider how they will ensure 
confidemialily of any information that can be used lo identify a user (e.g., name, e-mail 
address, postal address, phone number, etc,). This is especially important w'hen dealing 
with K-12 students since educators and parents may diseourage children from providing 
personal information lhal will be accessible lo all Intcmcl users worldwide. This concept 
is consistent with the American Library Association’s Policy on Confidentiality of 
Library Records (American Library Association, 1986) and the American Library 
Association Code of Ethics, w hich stales “We protect each library user’s right to privacy 
and confidentiality with respect to information sought or received and resources 
consulted, borrowed, acquired or transmitted’' (American Library Association, 1995 
Online). It is only appropriate that AskA services adopt core policies and philosophies of 
established library associations. (Sec Module 5: Contributing, 1‘or suggestions on ensuring 
user confidentiality.) 

Service Liability. Services rely on their information specialists to provide information 
based on expertise and knowledge. Services should make sure that users arc aware of the 
limits on inforinalion provided. This is especially important for services whose expert 
information can he interpreted as professional advice (e.g,, medical, legal, etc.) hut is a 
valid concern for all types of digital reference services. For example, an AskA service 
providing information on dental care should be clear in slating that the dental expert's 
response is not a substilulc fora consultation with the user's dentist. Similarly, library- 
based electronic reference services should make the same type of statement when 
rcfciring users lo dental information and especially when including any information taken 
from related information resources. (See Module 5: Contributing, for suggestions and 
examples regarding service liability.) 



Copyright. While .AskA service information specialists normall}' provide original cxjX'ri 
information or rcl’crrals lo information resources, there arc times when they include 
information taken from other sources. In wailing an AskA service response, information 
specialists should give credit to appropriate sources the way they would when writing 
other types of documents. While this issue is always important in providing information. 
It is especially crucial for services that provide public access lo queslion-answ'cr 
collections. 

Scr\ ices may also be concerned with co|)yrighling information on their own site (e.g., 
cjLiestion-answcr archive, information resources, etc.). While this is not an option for 
many services (especially those w’ith government sponsorship), other services can rely on 
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ihcir parent organizations’ copyright policies if available. (Sec Module 5: Contributing, 
for suggestions on handling copyright issues.) 

Openuiona! Issues 

Lack of Software. Currently, there is a lack of software available to assist AskA .services 
in managing the question-answer process. Some services have attempted to automate the 
question-answer process by developing original software packages (many based on PERL 
scripts). E.xamplcs of PERL-based software work-flow packages are MODERATOR from 
the MAD Scientist Network (Dry, 1997) and the Doctor's Office from Ask Dr. Math (The 
Math Forum, 1998). Other services manage the process using an e-mail program and a 
pencil and paper to record question routing activities.’ 

Staff Recruitment and Management. An AskA service can require anywhere from one 
to 500+ staff members. Various job roles include service administrator-coordinator 
(mantiging day-to-day operations and long-term decision making), e.xpcrts (responding to 
user questions; often \‘oiunicers), support staff (overseeing the work ofexperts; 
sometimes called moderators, team leaders, ele.), and technical staff (developing Web 
resources, software, etc.). These staff roles arc not always distinct; often one or two 
people will overlap se\’cral roles. However, these roles do represent the main functions 
involved in maintaining an AskA service. Issues relating to AskA service staff include 
recruiting, training, and quality control. 

Funding. Most AskA services operate on a not-for-profit basis, meaning that they must 
acquire funding from an external (or internal) source. Common sources include the 
government, professional organizations, universities, and corporations. In some eases, a 
service is an added function of a larger cxisiine organization (c.^.. Ask Shamu from Sea 
Woiid/Busch Gardens). In many cases, services must acti\c!y seek funding through 
grant-writing activities. Sec Wasik ( 1998) for suggestions and resources to help services 
seek funding. 

Marketing and Publicity. While some services openly embrace the opportunity lo 
attract users, others arc more hesitant for fear tb.at tlicy may receive more questions than 
they can handle. Difrcrent techniques for advertising a service include registering the 
service with a Web search engine, placing notices on other organizations’ Web sites, and 
posting messages on related clcclronie discussion groups, etc. (Sec Module 5: 
Contributing, for more suggestions and examples). 

‘ Romc(J> ('orpofiiiion is working with the Virtual Rctcrence Desk lo develop a eiislomi/ablc soriware 
package (o help individual Ask.A scrvicc.s manage llicir (|Ucslion-answcr and areliis ing processes. 
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Question-Answer Policy. Answering user questions is not always as straigluforward as it 
may sound. Decisions must be made early on regarding the following questions: Wluil 
types of questions will and v^ ill not be answ’crcd? What are the necessary components to 
include in a response? How will vague user queries be handled? What is the turnaround 
time for a response? Decisions on these issues will aid staff in conducting day-to-day 
tasks and will help services focus on intended goals. (Sec Module 2: Planning, for fuiihcr 
discussion on question-answer policy.) 

Supplemental Resources. Most services offer some type of Web-based resource to 
supplement their question-answer service; in some cases, it is the service that 
supplements the pre-existing resources. The most popular types of resources arc question- 
answer archives'^^ and collections of frequently asked questions, or FAQs. Other resources 
may include supplemental information about a popular topic and lists of links to other 
resources. Services often encourage users to review the collections first before submitting 
an original question. Issues related to supplemental resources include type of user 
interface, number of question-ansv\'cr sets included, frequency with which resources 
updated, and staff mcmber(s) responsible for resource maintenance. 

Dealing with AskA Sen'ice Issues 

Later modules of this Staner Kit wdl discuss many of these issues as noted, although 
there are no hard and fast rules for ensuring that these issues arc resolved smoothly. Case 
studies are included to illustrate ways that existing services have dealt with various issues 
in their experiences. 

Quality Characteristics and Standards 

Besides some common issues and trends, ccitain characteristics have been identified to 
determine the quality of an AskA service for the K~12 community. The Virtual 
Reference Desk Expcii Panel identified 12 characteristics and features (referred to as 
"facets of quality") for building a digital reference service for the K-12 educational 
community. The facets of quality arc based upon panel members’ experiences in 
managing and coordinating exemplary digital reference services. This list is intended as a 
set of standards for organizations to achieve in creating and maintaining digital reference 
services in ways that accommodate specific service requirements and characteristics. 



'"The Virluul Referenee Dci»k is developing a knowledge base t)t' quosiion-answer sets troni a consoriium 
of services lo enable users to search from a comprehensive collection of questions and answers before 
sending a question lo a particular service. 
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The facets are divided into two main categories: user transaction and service 
developmcnt/managcment. The user transaction category includes those components that 
occur during the question-answer process; these features are generally visible to the user. 
The service developmenl/management category involves decisions made in creating and 
maintaining the service that affect overall quality and user satisfaction. 

The 12 ‘"facets of quality” arc listed below (sec the following pages for a complete 
discussion]: 

Use r T rai i saci iof i 

1. Accessible 

2. Prompt turnaround 

3. Sets user expectations 

4. Interactive 

5. Instructive 

Sen ice Development and Management 

6. Authoritative 

7. Trained information specialists 

8. Private 

9. Reviewed 

10. Unbiased 

1 1 . Provides access to related information 

12. Publicized 
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The N'irUKil Kderenee l\'sk l:\jieri Panel has idem i Tied twelve ehiiraeierisiies and I'ctUiires (rel'erred l<i as 
laeeis uf quality) Idr huildinu a digital referenee serviee for the K--12 ediicaliniial eommumi)'. I'he facets nf 
quality are based upon panel members' experiences in managing and eoordinaiing exemplary digilal 
relerenee services. 'H'lis list is intended as a set oT .si and aids Idr organi/.aiioiis to aehieve in ereaimg and 
inainiaining digital referenee services, 

Tile facets are di\ ided into two mam categories: user transaction and service development ami 
iiKinagemcnt. 'I’he user transaelion eategory includes those components that occur during the question- 
answer process; these features are generally \ isible to the u.ser. The seiwice development and management 
eategory involves decisions made in creating and maintaining the service that affect oNcrall quality and usci- 
satisfaction. 

It IS important to note that K- 12 digital referenee ser\ ices dift'er t'mm each \Mher in .se\cral a.spccis 
including jiolicy and procedure, suhieei matter expertise, and suppoi img technolc'.gy. I*.aeii service should 
apply the rocommcndalions bciow in wa\s that aeeommodalc Us spccitic rcc|uircmcnts ami clKuaclcrisiics. 



Ifser Tran.sjiclion 
ACC’ITSSIBLI' 

Definition. K-12 digital rol'crcncc sciwiccs should he easily reachable and na\ igah!e by any Iniernci user 
regardless tif equipment sophistication. (Many users et^nncci from school-based compuler .systems that ma\ 
not iuive liigh bandwidth capabilities.) 

Stralegic.s. Digital reference services can ma\imi/c aeeessihihiy by: 

• Providing options for contacting the service (i.e.. offer e-niail address and Web form) 

• Designing a VV'eb interface ilial aeeommodalcs users w ith low-bandw idih eapabihiies (i.c.. 
minimal graphics and animaiioii). 

l>K()iMPT Il'RNAKODND 

Ddlnilion. All questions slunild be addressed as quickly as possible. Aetiial turnaround lime depends on a 
seiA ice's tjueslion-answer policy and available resources (e.g.. sialfing. funds, eie.i. 

Siralcgit's. Some exemplary digital reference services siiue to answer int]iiiries within two husiness da\s. 
while others plan to take longer (e.g.. up U> two weeks). 

SKVS VSl'M TAIMvCTATlONS 

Dcrinitiuii. ('lca»' communication should occur ciihcr before or at the start of every digital reference 
iran.saetion in oru to reduce opporuimlies for u.ser confusion and inappropriale inquiries. 

Strategics, Quest ion- answer procedures and services should he stated clearly either on a Web .Mte or in an 
automated ret|ucst aeknow ledgmeni to the u.ser. Siaiemenis should include the following items: 

• Time frame for returning response 

• I'ormai of response (e.g., what it will iiielude. where iespon.se will appear) 

• Rate of questions answered by .service 

• I {.Xpert t|iialifieations of: 

0 SciA lee 

0 'fhosc prov iding answers (information specialists} 

0 Rc.sourccs consulted to find answers. 
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INTKRACTIVK 

Dermition. Diglui! rcl'ercncc .scin ices shniild pm\ iJc uppiHiumlics I'ur users lo anvumimcuie neeess.iiy 
int'ormalion U) informaiion specialiMs and clanl\ nul’iic user questions. I'lie more opporuimiies lor 
iiiieraeiion, the more ctleeli\e tlie iransaetion. 

Strategies. Impoiiani inrornuinon - sueh as user aye or aradc te\el. oilier soinees cheeked, eoniaei 
inl'ornialion — can be captured tfmniyh the following mechanisms; 

• \\'el>- based request form 

• Interactive eomnumicahon tools le.g.. chai room). 

In addiliim, user r)ueslions ean he elarified ilirough iollow-up e-maii messages oi' coincrsalums using 
inleraciive eomnuimeations tools. 

i.NSTlU'( TIVK 

Definition. Digital reference sciwiees ean play an imporiani role in (he learning j^roeesses ol’ l^oth ehildicii 
and adults by proudmg access to eurrem informaiion and expertise. Qualil\ digital reference serueesolfer 
more to users than straight, factual answers; iiie\ guide them in siihjeei know ledge as well as the area o\' 
information hierac> . 

Strategie.s. hilbrmaiKui specialists can otTcr poinicis and paths used to find the best resources, so users ean 
learn lo answer similar quc.stions on ihetr own. I'or example, mfomiaium .specialists ean iiicmion (he lotds 
Used lo find resmiiees (c.g., search engines, indexes, bibliogiapliics. ealahigs), specific search lerms and 
processes used, and .senes ol' steps taken. 



Service Development and Manugement 



AUTHORIT.\'ri\ K 

Definition. 'Phe information sjieeiali.sis ol a digital reference service siiould Iia\e ihc ncees>ar\ knowledge 
and educational background In the service's gisen .subject area or skill in order to qualify as an expert. 

vStrategies. Specific levels of kiKwUetlge, skill and expenenee are determined by each service and its 
related discipline or field. 

TRAINKD INFORMATION SPbX'lAUS IS 

Definition. Services should offer cffeclive (u ientation or training processes lo prepare mforinatinn 
specialists lo resjiond to inquiries using clear ami elTeetise language and lollowing .seiwiee resj'umse 
guidelines. 'Praining of informaiion .specialists is one of the most importaiK aspects of planning and 
operating a digital reference service. 

Strategie.s. Many ptissihle training models exist for digital reference .services. 'Praining components include 
written documents containing response guidelines and opportunities for information specialists to praetiee 
responding lo inquiries accompanied hy fecdhack on pcrformanec. (Sec Kasow it/ 1 199S1 for background 
and informalimi on general skills involved in composing responses for the K-I2 eomiminity.) 

PRiN'ATF 

Definition. All commiiniealioiis between users ami informaiion specialists should be held in complete 
pnvac)-. 

Stnitcgie.s. Digital reference sees ices should receive eonseril from users before sharing transaction data or 
identification information (c.g., e-mail address) with a third party (c.g., questions and answers posted m a 
Web'hased archive). 

RFVIFWFI) 

Definition. Digital reference soiwiees shcuild regularly evaluate (lien processes and services (i.e,, 
responses). Ongoing resiew and assessment helps ensure ciua!i(>\ efficieney. and reliabilit\- of transactions 
as well as overall user satisfaction. 

Strategics, [''valuation meeliamsms ean include leview of inlormation specialists’ responses, .suiwey of 
users and feedback from informati('n specialists. 
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UNIMASLD 

Definition. Digital reference service.s should not prcmioio products or personal and institutional opinions in 
sucli a way that interferes with quality or use of service. Tliis is especially im[X'>rtanl because adults in the 
K-12 community (e.g., parents, administrators, teachers) may be concerned with children’s access to 
Internet-based information. 

StrateRie.s. Some ways to au)id bia.s in service responses and niatenal.s include the following; 

• Provide a balanced set of resources and provide statements of context for any type of opinion or 
viewpoint (e.g., “research states , . “s{ime people believe . . etc.). 

• When including information from another source, include full bibliograjdiic eiiation.s to emphasize 
that the information provided is not expre.ssing a personal opinion. 

• If eoinmcrcial advertisements must be included, they should not interfere with the primary 
information prtwided by the service (e.g., place product logos as smaller icons at the bottc^m of a 
Web page). 

PkOVIDKS .ACCESS TO RELATED INKORMATION 

Definition. Beside.s offering direct response to user questions, digital icferenec services .should offer access 
to supporting resources and information. 

Strategies. This can include infonTiaii<m on a service's special content area, access to a knowledge ba.se of 
previously-asked questions and answers, and links or references to external resources. Web sites should be 
revicw'cd and updated regularly to ensure that content is correct and links and references are active. 

PUBLICIZED 

Definition. Services pren-iding information Ui the K-12 community arc responsible for informing potential 
users of the value that can be gained from use ol’the .service. The greater the outreach to K-12 C(mimunitics 
across the country, the smaller the gap between the “haves” and the “have-nots” in terms of effective 
learning opportunities. 

Strategics: A well-defined public relations plan can ensure that service.s arc w'oll-publicized and promoted 
on a regular basis. Methods for publicizing a digital reference service may include: 

• Direct mail campaigns to potential u.sers 

• Promotional mcs.sages appropriate electronic di.scus.sion groups (e.g.. listservs) 

• References to the service on related Web sites (in the form of information as well as links) 

• A Web site of resources that points users to the qucslion/answcr service as well as supporting 
information resources. 

Conclusion 

K-12 digital reference services play an importiini role in leading students, parents, teachers, and others to 
information on and off the Internet, New' and cxi.sting digital rel'erencc services can carry out this 
educational role by using the ab(wc rccmnmendatiims as a guide in service dccisicms and question-answer 
transactions. 



^'|•om u Compiled by Abhy Kasowit/. Conicnl is bused on discu.ssion otA'iriual 

Rcfeicnec Desk lixpeii Panel members (July to Dceember 1997) moderated by David Lankes. Original panel members 
included Blythe Bennett of KidsComieci. Lynn Bry of the MAD Seientisl Network, Martha Dexter of the Library of 
Congress. Peter Milbuiy ol' LM_NI:iT, Joan .Stahl of the National Museum of American Art/Smilh.soman. Robin 
Summers of AskPRIC. Stove VVeimarofthe Math Poriim, and Ken William.s of the Math Loruni. 
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L3 Tools and Methods for Gathering AskA and Audience Information 

An organi/.aliun can use Informalion ahoui general trends in the digital rd'erence Held 
and eommon issues and characlerislies (n begin exploring the eiu'ironnienl in context ol‘ 
one’s own plans Tor digital reference sers'iee. This section discusses tools for researching 
existing services and methods for exploring potential user needs. This lyj)c ol’ research 
can help organizations decide whether or not to pros idc service or to focus on a specific 
area in terms of conlenl or nature of service. 

Gathering Information About AskA Sc‘rvices: I'he AskA+ Locator 

Organizations can research existing AskA scr\'iccs on ihcir own to find information 
relevant to their interests and intended direction of a jX)tenlial scr\ ice. One way to 
conduct this type of research is by searching ihc Virtual Reference Desk’s AskA+ 
Localor, a Web based collection of high-c|uality AskA services that focus on the 
information needs of the K-12 community. I'he AskA+ Locator can be found at 
h}tp.//\\'\\'\\\vnl.or^/U)cator/nulexJuml 

V.v: Locator was initially de\‘ck:pcd in the summer ol* 1997 as an informalion resource 
for both K-12 users seeking answers to questions and for AskA services requiring 
c|uestion answering support and resource referral. 

I\ncntial services were culled IVom a variety of sources, including resource lists such as 
Pitsco’s Ask an Lxpert site h}tp://\\'\v\\\uskau(\\pcrLcn}n/(iska}icxpcrt/\ federal agencies 
such as NASA, and othei* educational sites and organizations. Additional AskA services 
were located through the use of Internet search engines by using a variety of keyword 
searches. 

P.aeh site was reviewed for its target audience (educators, students, parents), the subject 
area, its applicability to curricular needs, and other criteria as set forth by the Virtual 
Reference Desk Lxpert Panel (see “Facets ol* Quality"). In some instances, iiUcr\'icv\ s 
were conducted with services to elicit more information regarding the nature of Ihc 
sciA'icc and its operations. 

The Locator is a dynamic resource: defunct services arc removed and now services arc 
added as they become available. An online submission form at 

lif!p://\\'\\'\\'.vnI.f)ri>/l()cafor/f()rf!Lhntil is available at the Virtual Reference Desk Web site 
so that AskA services can submit their information and URL(s) for possible inclusion in 
the Locator. Each submission is then reviewed and evaluated to ensure that certain 
standards of quality are met. 'Fhc Locator also lists only those AskA services that arc free 

•vU 
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lo ihe general public: Although there are se\'cral quality fee-based services a\ ailahle. the 
Virtual Reference Desk lists only those services that offer equal access to all users 
regardless of economic status. 

The Locator differs from other Web-based collections of AskA services in a lew’ kev 



• The Locator includes only those services that scr\c the K-12 community and meet 
some minimal quality eriteria as identified by the Virtual Reference Desk Expert 
Panel. 

• Detailed meta-descriptions are provided for each scr\’icc in the Locator. These 
“Profiles" include service identification information (e.g., e-mail address, contact 
person, links to services' home pages), scope, target audience, and a general 
description. Each profile links directly to the AskA service's home page and question 
submission page. 

• A search engine is built in lo allow users from the K-12 community as well as those 
researching AskA services lo locale services by keyw-ord, 

• Besides offering a search engine, the Locator prov ides subjcci and alphabetical 
listings of all serv'ices included. 

Some organizations may use the AskA+ Locator to find services in a particular subject 
area. For instance an organization interested in providing an astronomy Ask/\ service 
may want lo survey the field ofcxisling AskAs to see if there’s an aspect of astronomy 
that is not currently met by existing astronomv' services. Such an organization could use 
the Locator in this w'ay: 

Go directly to the subject listing, .K^-iL()r^/l()C(it()r/siihjcct.}UiuL tind 

scroll dow'n lo the heading “Science," subheading “Astronomy.” Here one wail 
find a list of 10 .services that already address questions dealing with astronomy. 
To learn about each service in more detail, one can click on the title of each 
service and go directly to its profile. The profile wall reveal such information as 

• question- “answerers" (e.g., asironomei's, astronauts, classroom teachers, 
etc.) 

• scope of subject areas addressed (e.g., space sciences, science careers, 
general science, etc.) 

• service eperation (e.g., answers all questions vs. representative sample, 
includes archive of previously asked questions, etc.]. 



Using this type of information, organizations can determine whether ov not (or to w'hai 
extent) ihev can oflcr a distinct, valuable service to the K-12 education communilv. 
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Tools for Surveying User Needs 

Besides surN'cying ihc cuiTcm collcciion of AskA services, il is impt^riani lo gaihcr 
information from or about the potential user eommunity to make sure that there is a need 
for a particular type of service. Ob\ iously user needs will be monitored on an ongoing 
basis once the service is implemented (sec Module 2: Planning and Module 6; 
Evaluating), but it is important lo establish an initial purpose for the serviee based on 
identified needs. For instance, an organization may discover through research on 
educational standards that national civics standards for high school students imply 
understanding the importance of participating in a democratic society, but there are \cw 
resources that help students apply democratic concepts lo their personal lives. This is 
good ammunition for an organization that is interested in setting up a ser\ ice lo connect 
high school siudcnls with ei\'ic organizations. 

Some methods for researching initial user needs regarding certain subjcci areas include 
the following: 

• Monitor educational or other appropriate c-maii discussion groups (iisiservs). ' 

• Provide a questionnaire on an organization's Web site or location. (For instance, a 
library contemplating an electronic reference service can provide a questionnaire at 
the reference desk to solicit feedback on the idea.) 

• Hold focus groups, inlcrN’icws, or synchronous electronic discussion with potential 
Lasers (c.g., teachers, students, parents, etc.). 

• Research educational literature and other rcscuirccs (Web, etc.) to establish 
educational trends in a particular subject area. 

• Research national or regional standards for education. 

Engaging in such activities and gathering user information alhnvs organi/.aiitMis to be 
iriorc prepared in the planning stages and assures that ideas for serviee arc \'alid and 
useful. 



‘ I-or a list (if some educational Iisiscrvs. see the .\skiq<lC' liducaiion I.isi^cr\ Aichoe 
hflp://(nkcric.or\»/\ '\ mud/ List sen _A nhivc/ 

A 
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Work Shee! 1-1: Survey of AskA Environment 

This page is available for you to record information as you research user needs and 
existing AskA service subject areas. The questions below are intended to facilitate your 
research and decision-making but do not account for all issues and questions that may 
arise during this process. Please feel free to record any additional ideas on a separate 
sheet of paper. 

AskA Service Characteristics 

L Service type: 

K-12 curriculum area: 

Other topic area: 

General reference 

2. Intended service users (check as many as apply): 

K-12 Students 

K-12 Educator/Administrator 

Parent of K- 1 2 students 

General Public 

Other: 

3. Stakeholders (individuals or organizations with an interest in service success): 

K-12 Students 

K- 1 2 Educator 

Parent of K-12 students 

General Public 

Other: 



User Needs Survey 

I. Survey audience (stakeholders): 



2. Primary survey questions; 
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3. Survey methods: 

Lisiservs, electronic discussion groups 

Survey (mail, phone, e-mail) 

Informal discussions with stakeholders 

Literature research 

Review' of academic standards/curriculum areas 

Other: 

4. Results 

There appears to be overall interest in proposed service. 

There appears to be some interest in proposed service. 

There does not appear to be any interest in proposed service. 

5. Comments (including ideas for alternate direction of proposed service): 



Ask A Service Areas 

1. Sources of existing AskA Services: 

AskA+ Locator liUp:/Ay\vw.vrd.(?rg/!ocator/si{hjccfJ}fml 

Web search (search by topic using Web search engines) 

People (K-12 community and others) 

Other sources: 

2. Results 

There is a significant number of high quality AskA services in area of 

proposed service. 

There are some high quality AskA services in area of proposed service. 

There are no high quality AskA services in area of proposed service. 

3. Comments (including ideas for altcimate direction of proposed service): 
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Module 1 Summary 

'rhissociion provided un iiUiodiiction to the field ol'diyilal reference, ineUidin^! of exisiiiiii AskA 

services, general trends and issues, and tools for researeliing existing AskA ser\ iees and potential user 

needs. 

• Digital referenee services (als(> referred to as .-\sk.\ scr\iees) are Iniernei>hased quesiion-and-ansuer 
seiA’ices that eonneei users with individuals who possess s|^eciaii/ed subject or skill expertise. Many of 
these Users are from the K -i?. edueatit>n ciminuinity te.g., students, edueatcM’s, parents, etc,). 

• Digital reference ser\ lee affects different communities in different ways: 

1. Members of the library eomiminity have instituted digital reference as a means of providing 
additional and more convenient access to resources for library patrons within and hevond the 
local comm unity. Digital referenee represents a change in traditional reference .service 
(librarians’ roles, access to informality, etc.). 

2. Members of the education community have henel‘ited from subject and skill experii.se 
provided bv Ask .A services. 

3. Members of the business aunmunity use vviuk-tlow software such as help desks and call 
centers to provide person-io-person connections w iih clients and employees. 

• Current quality AskA services focus heavily on the suhjecls of science and general reference but arc 
lacking in arca.s such as art, historv . literature, foreign language, and world geography. 

• AskA services face .several common issues including increasing question h)ad. liability, eont'ideniialitv , 
std’iware devcltqmient, funding, publicity, staff rceruitmeni and management, ptdiey develtqimem. and 
resource dcveUipment. 

• K-12 digital referenee services can etmsidcr 12 ’’laicets of Quality” in creating and maintaining 
service. 

• Inlbrniatitm aivnii existing .Ask.A services for K- 12 education can be found by using the Ask.A + 

Locator database of high-quality AskA services. 

• Inforrnalion on user needs can be gathered through a variety of methods, including eleeironie 
discussion groups, quesiitynaires. interviews, and literature research. 

I 

I 

I 

I 

I 
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AskA Overviews 

The following o\'erviews offer sarveys of ihc digital reference field in the subject areas of 
art and paleontology. They are provided as examples of research conducted to support the 
creation or ongoing work of AskA services in certain subject areas. These arc models of 
good AskA researcii. See ]v\\'\\\vr(l.()r^/AsLA/{li^c\sis.liiml for these and other "AskA 
Digests," brief repoas on cuirenl topics of interest to AskA services and users. (All URLs 
were current as of 1 November 98.) 



Report of Art and Art Education AskA Services 

Joann M. Wasik 
Virtual Reference Desk 
April 8, 1998 



The State of Art-Related AskA Services 

Despite the proliferation of AskA services, many subject areas remain inadcqiiatcK' 
represented with \'ew or no services in effect. For example, while users seeking 
information from scientists or reference librarians have several services from which to 
choose, those seeking art and art education information find very few seixiccs available. 
The Viiiual Reference Desk {hup://\\'\\'\\\vrcior^} has identified the following digital 
reference services that answer questions in different ail-rclatcd areas: however, the scope 
and capabilities of these services differ widely. 

Art and Art Education 



National Museum of American Art Reference Desk 

lutp://}uuLia-rydcr.si.ed{i/referciiccdesk/ 

This service both answers questions and pro^'iclcs information referral to its users within 
fi\’C business days. The scope is comprehensive and includes categories such as 
traditional fine arts, decorative arts and crafts, architecture and landscape design, 
photography, film and video, and commercial and graphic design. The ser\ icc provides 
information on American art only. 
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Art Studio Chalkboard 

hnp://\v\v\v.saiima^,C(ln/an/s{\ulio/cl\aikh(}ard.lU}}il 

This qucslion-tind-ansvvcr scrvitx funciions primarily as a resource for ariisis and ait 
siLidenis with questions regarding artistic technique (c.g., perspective, color, shading, 
etc.). Operated by a faculty member in SoulhciT) Arkansas University's art department, 
the service will occasionally answer art-related questions on other topics, but does not 
encourage (HU-ol‘-scopc questions. Qucsiic^n response time varies from several days to 
two months. 

.Metropolitan Museum ot ‘Art 

fiUp://\v\var.c()n]/c^i-hin/bbs/BBS/hhs_cutr(UU'CA:^i 

I'hc Absolutely Arts Visual Arts Forum is an online bulletin board/discussion list where 
users post questions to be answered by other users. Since this service is not a true AskA 
service m that any user may respond to inquiries, dilTering opinions may prove to be 
confusing for some users seeking information, and some answers may be of questionable 
\aliie. 

Other Art- Related AskA Services 

Bhoio^raphy 

Woir Camera & Video's Ask the Experts 

la{p:/A\’(ilfcaf/}cr(i.C{)tn/cyi-hiNAv{}lbc(}nf/l<)^in 

I'ormerly Online Photography Workshop's AskA Question Service, Wolf Camera and 
\'idco oflcrs forums on several areas related to photography and video, including 
cameras, darkrooms, and other photographic equipment. The service is for questions 
relating to j'lhotographic equipment and techniques, and does not provide infoiTiialion on 
jihotographers or photographic works. Most forums arc moderated, and many of the 
questions posted arc answered by Wolf Camera staff. Note: Wolf Camera's service is just 
one example of AskA services that answer photographic and photographic equipment 
q ueslions. 



Archih'i'lurc 

There arc several Ask an Architect sites on the Internet. Some arc fee -based services, 
w liilc others arc clearly geared towards clients. Some free services arc listed below. 

Ask an Architect 

hit p:/Av\\'\\\askiUiiircliiiccLcomAisk}nc. linn! 

I'his service is the volunteer effort of architect Parvin Gillim. Although the majority of 
hisquesiuMis come from users involved in “do-it-yourself' projects, he docs occasionally 

/ - ■ 

/ 

^ I 



34 




Inlorming 



Module 1 



answer questions from K~12 classes. Most questions receive replies within a week, 
although the service has had few inquiries of late; a faulty Web form for question 
submission may have been a factor in the recent lack of questions. 

Ask an Architect 

luip://infopoint.tl\eriver.c()ni/aiasa(:/ask 

Sponsored by the Southern Arizona Chapter American Institute of Architects, this online 
discussion forum appears to have been created as a resource for architects to share 
information with their colleagues. With eight messages posted since the forum’s 
inception 13 months ago, the forum is little used, and does not appear to answer questions 
from the general public. 

Conclusion 

With the exception of the National Museum of American Ail Reference Desk, and given 
the uncertain status of Ask A Question at the University of Auckland, there appear to be 
no other available resources for users with visuabarts-related questions at this time. There 
arc no available services, for example, to answer questions regarding European or Asian 
an. While several ari-rclatcd listservs and newsgroups are available to the general 
population over the Internet, any user may post replies to queries for information. As a 
result, any information obtained from these sources may be of questionable value. 

Due to the lack of digital reference services in subject areas such as art, the information 
needs of many users may be left unmet. Moreover, a potential burden may be placed on 
the few services that do exist in underrepresented subject areas, as the question volume 
may prove to be overwhelming for too few services to effectively manage. 
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Report of Paleontology AskA Services 

Joann M. Wasik 
Virtual Reference Desk 
June 28. 1998 

Dinosaurs and Paleontoloj^y on the Internet 

What is it about dinosaurs? The ancient reptiles flourished for 160 million years until 
their relati\ely sudden demise 6.^ million years ago. Yet the fascination with dinosaurs 
endures and continues to captivate the human imagination in children and adults alike. 

There is a wealth of information about dinosaurs on the Internet, ranging from tri\ ia 
about the Fiintstones’ pet Dino to the latest news on the links between dinosaurs and 
modern-day birds. Many of these sites receixe thousands of hits per day, as the public 
interest in dinosaurs and their habitats persists. Despite the extensix'C number of dinosaur- 
related Web sites, however, there are relatively few sites that ansxver user questions about 
dinosaurs and other paleontology-related subjects. 

The X’irtual Reference Desk [Iiiip:/Av\\'\\\vnl.nr^) has identified the follovx’ing digital 
reference services that ansvx'cr ciuestions relating to dinosaurs; howexer, the scope and 
capabilities of these services differ widely. It should be noted that services focusing on 
other paleontological areas, such as paleobiology, botany, and fossils, are not included in 
the folloxving resource lisi. 

Dinosaur AskA Services 

Ask-A-(ieologist (USGS) 

h/if ) ://^co I oi^y. usiis. i^o\ '/ask -a him! 

A scrx'ice of the United Slates Cicological Survey, this site ansxvcrs questions on all 
aspects of geology, including paleontology. Since this scrx ice pre.sently lacks the ability 
to keep subject-specific statistics, it is difficuil to estimate how many dinosaur inciuirics 
are rcccixcd and answered. 
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Ask Mike! 

l\ttp://\v\v\\\^iiunhc. cdu/-tkcesc }/dinoscinr/ashuikcJ\Wi 

Pan of ihc Dinosaur Web pages, this qucsiion-and-ansvvcr serv ice is hosted by a college 
student at the University of Maryland. The service cuirently receives between 3 and 10 
questions a week, and users may have to vv-ait up to six months to receive a reply. 

Ask the Paleontologist 

hup://\v\y\\'JyrrLdlnuiseH}}Lcom/p(iieo.ht}ul 

Slated to be unveiled in the summer of 1998, this new service is offered by the Royal 
Tyrrell Museum in Alberta, Canada. Questions will be answered by museum staff, and 
possibly international scientists as well. 

The Dino Mall 

lutjr//\\^v\vjnhx(miAVYOMlNG/D[NO/n^^^^ 

Part of the Wyoming Dinosaur Center and Big Horn Basin Foundation, this service 
provides answers to questions by the Wyoming Dinosaur Center geologists. The service 
currently receives an average of 6-7 questions per week, and answers l()07f in 1-2 days. 

1 users receive personal replies from the service. Although cuirently static, the site is 
undergoing a total redesign to be completed in fall 1998, The service expects .is question 
volume to increase significantly once the new. interactive site is launched. 

Dino Russ’s l.air 

hup://\v\vw.isi;,sjiiiic. cdK/is^srool/dinos/diuosJunuc.hhul 

Dino Russ’s Lair is a volunteer effort of Russell Jacobson at the Illinois State Geological 
Survey. The service receives approximately 3()-40 questions per week. Attempts arc 
made to answer all questions that require short, simple answers, and response time 
generally ranges from one week to one month, depending on Jacobson’s availability. 

Dinosaur Interplanetary Gazette; Dino Dish 

lUtp://\v\v\\\lusidcThc\Veh.c()m/messci^ehoard/n)hsxi*l/nih2976H 

Ho.sicd by an online maga/.inc of dinosaur news and information, Dino Dish is an online 
mc.ssagc board where users post questions to he answered by other users. Since this 
service is not a true Ask A service in that any user may respond to inquiries, differing 
opinions may prove to he confusing for some users seeking information, and some 
answers may be of questionable value. Dinosaur Interplanetary Ga/.ette also sponsors a 
special monthly Ask a Paleontologist message hoard service with notable guest 
paleontologists such as Jack Horner. 
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K-niail the Curator (The Worldwide Museum of Natural History) 

hllp://\v\\'\\\\\'mnh.C(jni/\vmvclOOUO.htm 

li-mail the Curator is a commercial site, which receives hundreds orquestions about 
dinosaurs from users each week. Only a few questions arc responded to due to large 
question volume. 

The MAD Scientist Network 

lutp ://)uadsci \ \-ustl. ech (/ 

MAD Scientist answers questions in many areas of science, including earth science. The 
service does not receive many inquiries for dinosaur information, and estimates that 
perhaps less than 17r of all questions received are dinosaur-related. 

Paleo Forum 

litip:/Av\v\\\pi[f.edH/--maifl/PatcoF()ntmInJbJiiml 

This service is a message board for discussions on paleontology, paleoanthropology, 
prehistoric archeology, evolutionary biology, and the evolution of behavior. Many of the 
topics on the discussion board lend to be rather technical in nature, and may not be 
suitable for many K~12 students or laypcoplc. As with ail message board-style forums, 
differing opinions may prove to be confusing for some users seeking information, and 
some answers may be of questionable value. 

SeienceNet 

ScicnceNct is a service that answers questions relating to science, technology, 
engineering, and medicine. At this writing, there arc approximately 60 archived answers 
to questions relating to dinosaurs. The service does not receive dinosaur questions on a 
consistent basis, but rather in “fits and stalls,” particularly after the release of dinosaur- 
related film productions such as Jurassic Park. 

Scientific American: Ask the F'xperts 

hf!p://w\v\v. sciani.com/aske.xpert/indcs' Jitnil 

An online companion to the popular print journal, Scientific American offers a question- 
and-answer service in several different areas of science, including geology. There arc 
very few archived question-and-answer sets regarding paleontology, and it appears that 
the service docs not receive many inquiries for dinosaur information. Inquiries as to the 
volume and scope of dinosaur questions received by the service have remained 
unanswered to date. 
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There are also dinosaur-related newsgroups and lisiscrvs sueh as the Dinosaur mailing list 
{dinosaur® use. edii) ai the University of Southern California, which works in conjunction 
with the Cleveland Museum of Natural History. Although paleontologists do respond (o 
user questions via the listscrv, public forums such as mailing lists cannot necessarily 
guarantee subscribers authoritative or timely replies to their queries. Other sites run 
special, time-limited services that are tied to specific events. To coincide with an 
expedition in the Gobi desen in 1997, the Discovery Channel Online hosted a “Dinosaur 
in the Dunes” queslion-and-answer service where users could e-mail questions to the 
expedition team. Question-and-answer sets from the expedition arc available at 
littp://\v\\'\\\discovery.co}}i/area/specials/i*()bi/e-mailrcpl\\IUfnl, although the service 
ceased operation when the expedition ended in August 1997, and no longer accepts 
questions. 

As illustrated in the above resource list, some AskA services arc overrun with requests 
for paleontology information while others receive relatively few questions. It is possible 
that users are unaware of the paleontology and dinosaur resources at some services; this 
seems particularly apparent in the case of services that answer questions in multiple areas 
of science, such as the MAD Scientist Network. Instead, users seem to rely primarily on 
sites that are explicit in their dinosaur focus, such as Dino Russ’s Lair and the E-mail the 
Curator service at the Worldwide Museum of Natural History. As a result, these latter 
services receive a high volume of questions and can answer only a fraction of the 
questions received. 

The demand for dinosaur information continues to grow unabated, and some AskA 
services have been forced to cease operations due to unmanageable question loads. The 
University of California (Berkeley) Museum of Paleontology discontinued their popular 
PaleoPals message board service in early June 1998 due to question overload and the lack 
of qualified staff to answer questions. As many inquiries required additional research by 
the paleontologists, and few questions lent themselves to becoming pan of a Frequently 
Asked Questions (FAQ) document, the service was unable to keep up with user demand. 

Digital reference and AskA services arc valuable resources that support the teaching and 
learning efforts of the K-12 community and beyond. As more services exceed their 
capacity to effectively and efficiently answer all questions received, many users’ 
information needs are left unmet. The creation of new AskA services, along with 
increased communication among existing services, may serve to promote shared 
resources and information. Such a cooperative effort may result in distributed question 
loads and increased rates of response to users, and thus provide more effective and timely 
information delivery, 
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Creating a Plan for Building and Maintaining 
an AskA Service 

Module Profile 

This module is designed to help organi/.alions design a plan for building and delivci'ing 
an AskA service that best represents organization goals. This module discusses issues in 
planning, including the general structure of AskA services, and illustrates through case 
studies the experiences of exemplary services. An AskA Plan work sheet is included to 
help new AskA ser\'ices organize necessary service components. At the conclusion of 
this module, you will he able to achieve the Itdlowing goal; 

(Joiil Determine the best way to build and maintain a digital rererence 

service within a given organization. 

Prerequisites Before de\'cloping and implementing an AskA plan, you 
should be able to collect information on the general digital 
reference field and existing AskA sei-\ iccs (see Module 1: 

Informing). 



Objectives At the completion of this module, you will be able to achieve the 

following objectives: 

1 . Identify service goals that arc consistent with those of 
s u pp o i1 i n g o rga n i za t i o n ( s ) . 

2. Stale components of general structure of AskA services. 

3. Create and implement an "AskA plan" for new ser\ ice. 

4. Apply general structure to individual service, 

3. Review plans of exemplary AskA services to identify 
important service components. 
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2.1 Creating and Implementing the AskA Plan 

The AskA plan is a documcni that outlines the potential components of a paiticular AskA 
service. The jdan is bused on a gencrali/cd structure of AskA services and rcflccls the 
overall goals of the ser\'iec and its supporting organizaiion(s). This sceiion di.seusscs 
issues in developing scrx icc goals, understanding the general AskA service structure, and 
implementing an AskA plan. 

Service (joals 

In order to create an initial plan for the development and implementation of a new AskA 
service, it is necessary to establish .service goals that arc consistent with goals of the 
supponing organization(s) (c.g., library, academic institution, professional association, 
government agency, etc,). It is impoilani to recognize that an AskA service can provide 
many benefits to its supponing organizalion(s). By promoting consistent goals. AskA 
services can serve as another vehicle to reach those goals. 

Digital Reference Service as Far! of Larger Organization 

Most digital reference services do not exist independently. They arc an integral part of a 
larger system: university program, professional association, government agency, etc. The 
digital reference component of an organization frequently operates under the 
organization’s mission, goals, and objectives. 

The organization and its digital reference service can share a symbiotic relationship. The 
digital reference service may rcccix c financial or operational support from an 
organization, although it is common (or a service [o seek outside funding and support as 
•well. The organization benefits from the relationship in important ways including the 
following: 

• Feedback to improve organization. By offering direct human interaction, a digital 
reference service can help the organization monitor user and stakeholder needs. 
Information from digital reference transactions can help the organization develop and 
impro\c resources and services (Lankes, 1995). 

• Promotion of national initiatives. Participation in digital reference service allows 
organizations and individuals to serve as volunteers through memoring in response to 
President Clinton’s call for increased voluntecrism (White House, 1997) and to carry 
out Vice President Gore’s online tutoring initiative to connect students to “a national 
network of top experts" (While House, 1998, Online). 
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• Impact on education. Digital reference service offers the oppoilunity to become 
directly involved in K-12 education by serving as an impoiiani component in student 
learning and professional development of educators. 



Wriling Scn^ice Goals 
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When writing service goals, the following questions should be considered; 

• Who arc the audiences of the service (both immediate and peripheral)? 

• What will members of the audience gain from using the service? 

• What will the greater organization and stakeholders gain from offering the service? 

• What will staff gain from participating in the service? 

One area that some services for K-12 students may want to explore is national or 
regional educational standards and benchmarks in order to target specific known areas of 
the cuiriculum''. 



Examples of Service Goals 



Below are examples of service goals as they may he stated for two different types of 
services. 



Goals of general subject Ask A service for K-12 students, where teachers serve as 
experts: 

1 . Motivate students to learn. 

2. Guide students in conducting research effectively. 

3. Help students improve written communications skills. 

4. Help teachers become proficient Internet users. 

Goals of electronic library reference service for K-12 students and adults, where 
professional librarians respond to user questions.'"' 

1 . Provide quality information service and resources to the public. 

2. Create quality information resources on the Internet. 

3. Promote the roles of librarians in the information age. 



' I'or informaiion on educational .standards, see “C(nitcnl Knowledge" from Mid continom Regiimal 
liducntional Laboratory (McKEd.) hitp://\v\y\v.nirrci.ori^/.sran(lar(Is-heiirlimarks/ and "Developing 
Hdueational Standards" from Putnam Valley Central School.s 
htt p;//\\'\\'\v.piit\vcst. bores. or^/Stcmdanls.himl. 

'' Al.so sec 'rhe Internet Public Library Mission Statement ht!p://\v\\'\y,ipl,or^/ahoiu/tie\vinission.hnnl 
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These types oTgotils can help guide llic digital reference service in decision making and 
development ol’ the service. However, a list of goals is only one of many components in 
the AskA plan. The following section discusses a general structure of AskA services that 
can be used to guide new services in establishing components within a plan. 

AskA Anatomy 

After the main aoals are identified to help guide the creation and management of the 
service, decisions should be made regarding plans to reach the goals using the suggested 
AskA framework as di.scusscd below. The AskA Plan should outline the planned 
resources, activities, and products of the service based on this general structure of AskA 
services. 

(jcncrcd Cnmponcnis: I'hc AskA Anatomy 

All AskA services can be described in terms of a conceptual framework with three main 
components: detectors, rules, and effectors. Bach of the three components represents the 
many roles, relationships, activities, resources, products, etc. involved in building and 
maintaining an AskA service. The components are described and outlined below. 

This siruciLirc is based on research of six exemplary digital reference services.'" See Table 
2-2 for a more detailed breakdown of components. 

Detectors. Detectors arc the organization’s mechanisms for acquiring inldrmation on (he 
environment. The knowledge gained through detector activities can help ensure that the 
service meets user and stakeholder needs and ti.ses the most efficient proccs.scs and tools. 
Detectors can gam information about the Ibllowing: 

• Users — general public, specifically members of K-I2 community (students, parents, 
educators) 

• Other Internet inlormaiion services — existing AskA services, general Internet 
information services, etc. 

• Application builders — developers of software programs that run ou the Internet (user 
interfaces, etc.): Web browsers, servers, other software (e.g., animation, real-time 
audio, etc.) 

• Infrastructure providers — technology used to make network connections (leased lines, 
bridges, routers, modems, etc.), standards regarding technology, etc. 

• Internal sources — service staff, participants (e.g., information specialisls/experts, etc.) 

• Bxternal sources — funders, ovei seeing organizations/inslitutions, etc. 

P’or nv)ic i-omplctc di.scussion of .scr\icc slnicturc and related rcseareli. see I.ankes ( 199S). The 
terms detectors, nilc.s, and effectors used by I.ankes ( 199S) were Je\ eloped by Ji>hn Holland of the 
Unisersily of Miehigan. 
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Informalit^n from these sources helps form the AskA's ‘‘vieu' of the world'' and forms the 
foundation for all decisions about how the scr\ icc is built and maintained. 

Rules. Rules represent the actions of an AskA service based on the information from 
detectors. Types of rules include question-answering, routing, archis'ing, resource- 
building, etc. Since rules arc abstract w’ilh no physical structure, resources must Ik' used 
to actually carry out the actions. Resources — people, software, hardware, etc. — arc the 
means of implementing rules. 

Human Resources. While the number of staff or volunteers in a scr\ icc can \ ary (as do 
specific job titles and number of people performing each task), there are .scvcial job roles 
or tasks that arc common to many Ask.A services. It also \ aries as to which roles {if any) 
are ))aid positions as opposed to volunteer efforts. Below arc some common roles or tasks 
and some titles a.ssociatcd with them: 

• Vlanage the daily operations of ser\ icc (c.g., administrator, coordinator, technical 
staff, programmers) 

• Institute policy and research areas for further dcN cloimicni {e.g. director, 
administrator, coordinator) 

• Respond to user queries (c.g.. information specialists, experts, \()luniecrs) 

• Oversee performance of those who respond to queries (if different from titles 
mentioned in 1 and 2, can be mentors, trainers, team leaders). 

Technical Resimrees. These types of resources include Web serxers and electronic mail 
foi' cairying out question-answering, archiving, and other processes. Table 2- 1 shenvs 
examples of technical resources as they coircspond with service rules. 

Table 2-1 Service Rules and Technical Resources 



Rules Technical Resources 



E,slabli,sh Web pre.scncc sei vcr space, eonncciion. eic. 

Receive incoming que.stions and feedback ti\mi e-mail, Web form 

11 sens 

l^espond to users can be individual e-mail account or W'cb page 

Rome questions to potcMitia! respondenis database, e-mail account 



Archi\’c previously asked questions 


daiaba.se. .soarcii engine, etc. 




Make aN'ailable supporting intbrmalion resources 


Web page (can include search engines, graphics, 
etc.) 





RITector.s. Effectors arc sets of services offered to u.scrs or tUhers on the Intel net in order 
to mcei reference needs. This can include the ultimate cjucstion-auswer service, 
supporting information in the form of a Web site (archixes, frequently asked questions 
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[FAQsJ, references to other resources), phone service, special events, etc. Effectors are 
the output of an AskA service. These mechanisms are an organization’s means of 
changing the world around it — or at least its own place in it. 

Table 2-2 outlines the three components — detectors, rules, effectors — in terms of their 
subcomponents (detectors are broken down by agents of information) and provides 
examples based on practices of existing services. The examples do not represent a 
complete sample of AskA components, but provide some ideas for organizations getting 
staned. 
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Table 2-2 AskA Service Structure and Common Components 



Detector 









effectors 



AR6nt/Subconiponent Examples*' 


Users 


• User surveys/feedback forms 

• Archive of e-mail interactions 

• Web server logs 

• Focus groups 


Internet Information Services 


• Web browsing for other Internet 
in lor mat ion services 

• F'orma! review, collection, and analysis 
of data 


Internet Applieatiim Builders 


• 'Trade journals 

• Web sites \sith software reviews, 
information 


Internet Infrastructure Providers 


Information and experience with 

• Bandwidth 

• Routing/switching 

• TCP/IP 


Internal Influences 


♦ FmpUwee ideas 


h-xierna! Influences 


• T'unders 

• Marketing concerns 

• Needs of larger organization 


Detector Information Process 


• Prioritize incoming Information 

• ITanslatc information into process (via 
meetings) 


Process 


• Question ansvsering 

• Triage of questions 

• Archiving quest urns and responses 

• Modification/development 


Resource Types 


• I lardware/softwarc tools 

• People 

• Policies 

• Information resources (used in question 
answering) 


Resource Types (people) 


• Roles (administrativc/question-routing, 
question answering, technical, research, 
etc.) 

• Skills 



EtTfcclor Tic- 1 ns 



Technical 



Policies for modifying existing 
scrviccs/crcalinii new services 



Website — question form, archive, I'AQ 
Ciopher site 
in'P site 
F.-mail service 
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Mda-Dc'scripfion 



From looking at exemplary digiuil reference services,'' the basic si rue I Lire of Ask A 
services can be described in a single absiraci incla-descripiion as seen in Figure 2-1, This 
figure follows a blueprint format and may be read left to right. The leftmost column 
represents detectors (Question Acquisition, Web Surfing, Publications, Parent 
Organization). These are mechanisms universally used by exemplary services to gather 
information for an AskA service. A single question-answer process is used to process 
incoming questions and is outlined below. The single effector listed in this abstract meta- 
description, "Answer Sent to User \ ia E-MailT is inverted. (Sec Appendix A for sample 
blueprints of exemplary AskA services.) 




'' See companion research report ‘'Building and Maintaining Internet Information Services: K- 12 Digiial 
Reference Services'* available froin 1{RI(* C'learinghouse on Information and 'rcchnology (Lankes, 1998), 
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This simple process covers the range of possible aciix iiies rclalcd lo qucslion-answcring 
processes; this docs noi represent Web dcvelopmcnl activities, as they arc common 
1 across the exemplary services researched in the companion research study. It also 

incorporates common resources and detectors from tiic exemplary services. Table 2-3 
[ defines the steps in the "Question-Answer Process" of the mcta-dcscription. 

I Table 2-3: Components of the Mcta-Ocscription Qucsaon-An.s\ver Process 



Step ■ Description y ' 



Quc.slion Acquisiiiun An c-niail address or Web f(’>rm is used lo allow a user lo enter a quesiitm. This 

may require the user lo determine a topic for the question. Some automated 
preprocessing ma\ occur lo put the question in HTML or assign .some sort of 

tracking ideniiricali(Mi. 

Pool ol i\)ssible Respondents .*\ mes.saue is queued and prepared fore-xperi rcspt)nsc. In scr\ iccs \wih multiple 

experts axailablc, some triage process occurs (for example, in Ask .-\ 
X’olcanologisl a Pl:RL .script simply sends questions to experts based on the 
daxs olThe week, while MAD Scientist Network uses moderators in .select 
experts). In single-pei.am operations, message.s are simplx queued lor (he single 
expert. 



lyxpert Answer Is CJencraied .An expert generates an answer. In some .seiwiees iliis is done in accordance wnh 

a policy document. In siune sciwiccs these answers come from Imcrnet 
resoLirees. and in all ca.scs the expert has personal knowledge of the topic. 



Answer iieni in User \ ia 
Ii-MaiT i 


Once an answer is generated, n is sent to the user \ ia e-mail (if possible). 


Tracking 

i 


Tivnd.s and subjects Irom questions arc tracked and u.sed, In some eases the 
trends are u.sed in \S'cb dcxclopmeni. Tracking can consist of Web-ha.wd 
arclioe. pri\aic archive, or simpK intoi'mal mfiirmation. 



Simply put. a question is rcccis'cd through the Web ore-mail. A process determines the 
best expen to answer the question. Once an appropriate expert has been identified, that 
expert formulates an answer. The answer is sent lo the user via e-mail. After an answer 
has been created a tracking process occurs. In .some cases this invoK cs the creation of a 
public Web-scarchabic archive. Other scrx'iccs use a prix ale archixe. Still others use a 
more informal tracking process to inform the creation ol' Web resources. 

Implementing the Ask A Plan 

Once the components are outlined and a general plan is formed, the ser\ ice can start 
implementing the plan: establishing processes, hiring staff, acquiring resources, etc. It 
should be noted that many existing ser\ ices hasc entered into operations (rules) and 
sciA'ice provision (effectors) without mapping out an original detailed plan. While this 
approach can result in quality scr\ ice. a new .service can sa\e lime and make more 
consistent decisions w'hen guided by a genera) structure as outlined in this module. 
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2.2 Creating Your Own AskA Plan 

This section offers i\ work sheet with a list of questions to guide new services in creating 
an AskA plan. Using the questions from the work sheet, the meta-description can be 
tuned and expanded to meet the needs of your organization. 
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2.3 Examples and Experiences in Planning AskA Services 

This section offers case studies of three exemplary K-12 digital reference services (Ask 
Dr. Math, MAD Scientist Network, and National Museum of American Art Reference 
Desk) in order to provide ideas and help organizations get started on their AskA Plans, 
Appendix A includes ‘'blueprints" of each service’s framework in terms of its detectors, 
rules, and effectors. The case studies and blueprints highlight K-12 AskA services that 
are considered top quality according to criteria established by the Vinual Reference Desk 
(see “Facets of Quality" in Module I). 

Planning AskA Services: In Their Own Words 

The following case studies describe experiences of three exemplary digital reference 
services. An administrator from each AskA service, offers information and advice 
regarding issues in the initial planning process as well as ongoing service development. 
Blueprints for each service (as described above) are included in Appendix A. The case 
studies include the following: 

• “The Evolution of Ask Dr. Math" by Ken Williams and Steve Weimar describes three 
phases of growth over the more than three years of math reference service. 

• “Getting Underway: I'hc Virtual Reference Desk at the National Museum of 
American Art" by Joan Stahl discusses the history, mission, and usage trends of this 
art information service along with lessons learned after five years of service. 

• “Planning an Ask-A-Scientist Service" by Lynn Bry shares questions faced during the 
development of this science question-answer service for K-12 students. 
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The Evolution of '’‘Ask Dr. Math” 

by Ken Williams and Stephen Weimar 

The Math Forum’s question-and-answer service, “Ask Dr. Math,” has undergone a great 
evolution since it began in 1994. From its inception through April 1998, Ask Dr. Math 
has taken on over 225 volunteer "doctors” from all corners of the globe. The service 
continues to grow in popularity and has received a number of Internet awards. Our 
“About Ask Dr. Math” page provides more information: 

http .-//fonon, swarthniorc.edu/dr.math/abt.cIrmaihJitml 

Our specific evolutionary process may be of interest to people who w'ish to start their 
own quesiion-and-answer services. In general, our growth can be divided into three main 
phases of development. These phases reflect periods of self-definition, expansion, and 
enhancement. 

Phase 1: Launching the service, evolving standards for quality 

(November 1994-December 1995) 

When we launched our service in November 1994, a small group of Sw^arthmore College 
students volunteered their time to answer math questions received from K-12 students 
around the world. We didn’t advertise our services very widely, since our staff was small 
and had limited lime. Initially we simply posted the following announcement to several 
math-oriented discussion groups on the Internet, including “gcometry-pre-college" and 
“sci.math”: 
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If you are a student in elementary, middle, 
or high school, v;rice to us ! • We can't wait 
to help you with those really tough or 
interesting problems. .^11 of the Ask Dr. 

Math letters are ansv;ered by members of “The 
Swat Team, " math students and professors here 
at Swarthmore College (and some famous 
"ringers" from elsewhere in the math world, 
including John Conv/ay at Princeton). Ask Dr. 

Math is a project of the GEOMETRY FORUl'i, an 
:jsF- funded program housed at Swarthmore 
College in Swarthmore, Pennsylvania, USA. 

Dr. Math is not a mai i ing- 1 i s t , it's an e- 
mail address chat you can v.’rite to v;ith math 
questions, and we'll write you back. if you 
'write to us, please include a subject in your 
message that says something about your 
question . 

At this Stage, Dr. Vlaih was just an c-mail alias: any mail sent to 

(lr.})icith@fonun. swarthmore. edii was forwarded to the entire list of \oluntcers. When wc 
answered questions, we copied our responses to the list. 

During this phase, we kept certain key questions in mind about how we wanted the 
service to grow' and develop. Some of the more important concerns were the follow ing: 

1. What kind of service do wc w ant to be? A homework help ser\'icc? A reference 
service? An iiueracti\‘e tutoring seiwicc? 

2. What distinguishes a helpful answer from an unhcipllil onc'.^ 

3. Will we try to answer all the questions wc gel? If not, w'hich ones will wc choo.se? 

It has been quite useful to keep these questions in mind throughout the c\‘olution of our 
,scrvice. The answers to these questions could determine very different paths for a 
question-and-answer scr\*iee. For instance, if a project chooses to he a liomcwork help 
service (w'hich is NO!' what wc chose as our primary mission), the project's highest 
priorities should probably include immediate answering and coordination with teachers. 

i < 
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Phase I Queslions and Ansnrrs 

You may want to look ai some inicraciions from ihc first phase of Dr. Math: 

i]ttp://f()nuu.s\vanh}}wn\cdii/di\nmih/pi'()hlc}}]s/l jyrimedifnd 

lutp://forw}).s\\'artlwiore. edu/dr.matli/prohlcans/pijrrationaljwndyers. hnni 
hnp://lonufi.s\var!Jimorc\edi{/dr.ma!h/prohletns/purp()se__zeradit}ul 
lutp://foni}n.s\\'(irdimore.edu/d}\niati\/pwhle}}\s/addition,!Uiul 
http'J/fonmLswardnnore.edu/dr.math/problcins/addin^jyctsduml 
http://fonn}].s\vardwiore.edu/dr.}uath/probUn}\s/s()!vi}]^Juird_jva\\ hwd 
http://foniuLS\\'art}uuore.edii/dr.math/pvohle}us/circle_chonlsd]iml 
IUfp://formn.s\\'arilimorc'.edu/dr.mafh/prohle}ns/c()lliiicardinnl 

Phase 2: Building an archive, incorporating outside volunteers, implementing 
standards 

(January 1996 -March 1997) 

It didn't take long for us to realize that a great many people had math questions. We 
needed to find a way to expand the scope of our service. To do this, wc asked ourselves 
other questions: 

1. What do we like most about our service'.^ 

2. What technical aspects of our answering prtK'css need to improve? 

Wc also began to develop answers to some of the Phase 1 questions concerning the 
service's character and issues of quality as we gained experience vv'ith our service and 
learned more about the needs of the community wc served. These answers helped us 
choose directions in which to expand. 

Qnestum-Answer Mana^enunit 

First, we needed to improve coordination among the volunteers and to save their personal 
I e-mail accounts from the deluge of Dr. Math-related messages. To accomplish these 

goals, we w'roie a software package called The Doaor's Office in the summer of 1995 to 
I manage the flow of questions and answers. This Web-based package included a ’Triage" 

* area where our volunteers could choose a question to answer, a "Post-Op" area where 

volunteers could review each other's work, and environments to view and answer 
I questions. The software sent answ'crs to users via e-mail. 

I 

I 
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lucorporatin^ Oiaside Volunteers 

Second, we decided that in order to answer more questions and broaden our range of 
expertise, we needed to enlist the help of volunteers from outside the Swaiihmore 
College community. To do this, we had to develop a training and review process for our 
new volunteers. We based this process in a new area in the Doctor's Office called the 
"Holding Tank,” where answers from our volunteers-in-training could be reviewed by 
more experienced volunteers before being sent to the questioner. 

Defuiing the Sen^ice 

We decided that we vvould not be a homework help service, because homework problems 
typically need to be answered within a matter of hours. Our project relied on volunteers 
to answer questions, and we didn’t want to ask them to be "on call” to answer questions 
immediately. We also decided that our primary role was not that of a reference service, 
since several good mathematics reference services already existed on the Internet. Also, 
our volunteers seemed more interested in actually doing math problems and helping 
students with math concepts than in referring them to other sources of help. 

the Archive 

We realized that simply increasing the number of volunteers wouldn’t be enough to 
combat the growing number of questions. Furthermore, our project was administered 
primarily by volunteer Swarthmore students who w'ould not have time to train all the new 
volunteers needed. We also realized that we were beginning to build up an archive of 
mathematics information that had value in its own right and that we could take advantage 
of this resource in answering questions as w'ell. Thus we decided to shift our focus to 
building our archive of previously answered questions. 

Once we made this shift, we realized that it made sense for many reasons. First, many 
people asked questions that had already been answered well and were in the archive. If 
they could find and understand an appropriate answer in the archive, they might not need 
to ask the question at all; in fact their needs would be filled more quickly by looking in 
the archive than if they had to wait for a reply. 

Second, many people benefit from seeing several different approaches to a problem. A 
typical response from a volunteer explains one way of doing a problem or may give just 
one example, w'hcrcas the archive can present many different views of similar que.stions. 
Exposure to a variety of approaches- leads to a greater understanding of the mathematics 
and often to a realization that mathematics is not primarily a matter of finding the right 
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recipe of calculations to perform. Indeed, vve fee! that one of the most valuable aspects of 
our service is that we can provide a very different way of looking at a problem than a 
student’s teacher might. 

Third, the archive allows us to focus more of our energy on questions we think are 
interesting, rather than answering the same kind of question many times. This helps our 
volunteers remain enthusiastic about participating in our project, and as a result they stay 
aboard longer and write more interesting answers. 

During this emphasis on the archive, we were presented with some technical challenges. 
We needed a smooth process for generating and editing an archive and an intuitive means 
of accessing it for both the volunteer doctors and the public. Conceptually w^e had to 
design a structure I'or the archive that would meet the needs of a wide range of users. We 
chose to organize resources by grade level, and then within those levels by math topic, 
thinking that this structure would be most familiar to our young users. 

Implementing Standards and Training Resources 

When we had airived at a more prcci.se definition of our service, we were able to create a 
guide for our volunteers that helped them write responses. The guide contains technical 
information about making everything work properly as well as guidelines for writing 
clear, helpful responses. This has been a wonderful resource for our volunteers. 

Phase 2 Questions and Answers 

Here are some interactions from our second phase: 

http://f()rnm.s\vurtlunore.edi{/dr.m(ith/prohlenis/hiimphreys6,22.96J}t)nl 
http://forum. swarf }imore,edu/dr.matli/prohlems/tao} 0.26.96.html 
Iittp://fonim.swarthmore,edu/dr.niath/prohlems/(leyo5.1 5.96Jitm! 
http://forunLswarthmore.edu/dr.math/prohlems/nnknown4.I7.96.lumI 
http ://f)rum. swarthmore.edu/dr.math/prohlcms/georgia4. 19. 96.html 
http './/forum.swarthmore.edu/dr.tfiath/problems/milhurnS. 29. 96.html 

You may wish to compare these with the interactions from the first phase. In general, you 
may find these more comprehensive and representative of more points of view. The tone 
of these responses may not be as enthusiastic as those of our eager college students who 
had the sense that they were helping create a new service rather than helping sustain a 
service. 
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Phase 3: Adding value to the archive and planning for the future 

(April 1997-prescnl) 

From the first two phases, we hav'e idcnlificd issues that guide our work and lielp us 
choose our priorities for future development. These priorities include the following: 

• Facilitating mathematical satisfaction 

• Managing increasing numbers of questions 

• Maintaining a consistently high quality of interaction 

• Leveraging the interactions so that many people may benefit from them 

• Impros'ing ihe question-submission process 

• Maximi/^ing the quality and use of our archive 

During Phase 3, we arc cuiTcntl\’ concentrating on adding value to our archive, educating 
users on how to effectively use the service, and creating new resources and processes for 
the future, 

Achliii^ Value to the Archive 

To add value to our archive, we have assembled several Frequently Asked Question files 
that prov ide an overview of various topics (e.g., PascaFs Triangle, Pi) and provide links 
to good responses on these topics. Unlike our archive, these FAQ files arc not organized 
around specific math problems; rather, they give richer, more comprehensive treatments 
of broad topics. FAQ files arc created by collecting and organizing a mimbcr of different 
questions and answers as well as other related Internet resources. Our FAQ is located at 
http://fari{}}i.s\vartlim()re.edii/dr.}}iath/f(ui/ 

User Education (Commiaucatin^ Ejfeetivc Use of Service) 

In managing the volume of questions we realized that vve needed to improv’c the entry 
process, focusing on answering questions that arc genuine, weil-formcd, and not readily 
answered elsewhere. To this end vve now provide recommendations for teachers who 
want to use our service in a classroom situation and tell them what kinds of things are not 
considered effective use. Since vve began providing these recommendations, 
inappropriate use of Dr. Math by teachers has decreased. Our recommendations arc at 
http://fo rum. swart I wiore.edu/dr.math/approp. use.html. 
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Planning for the Future 

To furlhcr expand die capacity of our service, we have cxpcrimcnled with ways of selling 
up mentorships between new volunteers and more experienced '‘Math Doctors." Until 
now, most of our training has been done in a ccntrali/.cd manner with our administrators 
mentoring our new volunteers. The time required to interact w ilh trainees has been our 
biggest bottleneck in terms of involving new \oluntecrs in our project, and wc would like 
to find ^JTective alternatives without compromising the quality of our service. 

We arc also considciing possibilities for “franchising," allowing others to set up local 
versions of Ask Dr. Math that have access to our experience, software, and central 
database. For example, a school district may want to set up a qucstiun-aiid-answcr sciwice 
in which students in llie high schools answer questions from students in elementary 
schools. 

As wc continue to focus on supporting individual students interested in exploring and 
learning mathematics, wc realize that the shoil, discrete questions and answers in the 
archive do not lend themselves to more extended investigations of topics. It is impoilant 
that ,scrvices such as ours he able to present their archive not as collections of discrete 
questions and answers but rather as more structured resources that present the inherent 
linkages and structure within their subject areas. Our FAQs help somewhat in this area, 
but we still feel that more support is needed. 

Towards this end, one of our most talented volunteers is w'orking with our support to 
develop a more coherent and multidimensional environment in which to explore math 
topics w ithin the K-12 curriculum. This environment would allow' a user to view' several 
different perspectives on a given section of the Ask Dr. Math Web site (c.g., fractions): 
different math concepts w'ill be explained through voices of teachers, peers, 
mathematicians, etc. This new resource wall become an integral part of the Ask Dr. Math 
environment so that volunteers and questioners can use it as a source of information, and 
anyone using it can be connected to the question-submission form. The question- 
submission process will capture information regarding the specific resources (i.c., Web 
page) used by the questioner when submitting the question. This contextual information 
will enhance the processes of answering questions and managing the incoming queue. 

Finally, wc would like to create a question-asking environment in which people's 
questions arc correlated programmatically with our archive, and submilicrs arc directed to 
appropriate archive entries. There arc many different w'ays wc could implement this, and 
our challenge will be to find out w'hich wavs arc effective. For instance, submilicrs could 
type in brief summaries of their questions, indicate which of our categories they might 



61 



Planning 



Module 2 



fall into, and press subinir. The next page will include several items from our archive that 
match submitters’ questions most closely. If they still want to ask their question, we will 
provide a form in which they could record the complete question. 

Phase 3 Questions and Answers 

Here are some examples of entries in our FAQ, including links to questions and answers 
from our most recent peiiod of activity: 

liup://fonnn.swanlimore.edii/dr.math/faq/faq.order.operations, html 
hUp://forum.swartlunore,edii/dr.math/faq/faq.n€gxneg.lit}nl 
hup ://forum.swarthmore.edu/drjnaih/faq/faq.sqrt,b\\iKind. html 
http://forunLswarthmore.edu/dr.math/faq/faq. comb.penn.html 
http://fonwLSwarthmore.edu/dr.math/faq/faq.doiibling.pennies. html 

Conclusion 

As we look toward the future, we look to create new ways for users to interact with the 
Ask Dr. Math resources and experts in order to motivate learning and support basic goals 
of K-1 2 math curricula. With our rich set of resources and over three years’ experience in 
answering student questions, we are confident that Ask Dr. Math wi,. continue to support 
math education and pave new roads for digital reference. 



Ken Williams, ken@forum.swarthmore.edu, is programmer and administrator for The 
Math Forum, and Stephen Weimar, steve@forum.swarihmore.edu, is the co-principal 
investigator of The Math Forum. 
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Planning an Ask-A-Scientist Service 

by Lynn Bry 



Introduction 



The MAD Scientist Network (MadSci) hnp://\v\\'\v.madsci.or^ is a Web-based “Ask-A- 
Scientist Service” started in 1995 by Lynn Bry and Joe Simpson, both M.D./Ph.D. 
students at Washington University Medical School in Si. Louis. The site began us a 
branch of the “Young Scientist Program” (YSP) {medicine .\vustLedu/-ysp/) , a student- 
run program at the medical school that aims to increase the science literacy of local K-12 
students through hands-on learning. We created the site to provide a means by which 
students at the local public schools could ask questions of the faculty, staff, and students 
at Washington University. Though our intentions were local, we were well aware that the 
nature of the Web provided worldwide exposure. However, we felt it best to initially 
target our efforts to a specific audience. More than two months of planning went into 
devising and creating the infrastructure for the site before it officially appeared on the 
Web. During this period the following questions were considered and addressed: 

What technical resources would be needed? 

Given that the service would exist on the Web, Web space was the primary requirement. 
The medical school provided Web space free of charge, including the ability to run CGI 
scripts and search engines specifically for the site. 

Secondarily, personnel would be needed to create Web documents and CGI interfaces 
and to oversee the distribution of questions among experts. HTML and CGI scripting was 
undertaken by Lynn Bry, while Joe Simpson oversaw the basic administration of the site. 
We felt that CGI scripts would greatly facilitate the utility of the service by making 
information searchable, assisting the selection of scientists to answer questions, and 
assuming many of the time-consuming, mundane tasks such as formatting information 
with hypertext and generating listings of questions and answers. 

Who would answer questions? 

Forty scientists at the medical school volunteered to answer questions in response to 
fliers posted on campus. Their level of education ranged from full professors to graduate 
and medical students. An online form at the preliminary Web site allovscd people to sign 
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up. The form asked for basic infonnalion as well as the areas in which people wished to 
answer questions. 

What areas of science would be targeted? 

We based the areas on the primary areas ofexpenisc at the medical school — the 
biological sciences and chemistry. The areas were divided into 20 "categories" (c.g., 
anatomy, biochemistry, chemistry, genetics, medicine, zoology, etc.). 

How would questions and answers be received? 

During 1995 many ol' the St. Louis schools had or w ere expected to have Internet access. 
Receipt of information through Web forms seemed the easiest and least labor-intensive 
method of receiving questions and answers. Information from Web forms could be easily 
processed by Perl CGI scripts to create HTML files. In this manner both questions and 
answers could be easily presented on the Web, providing an easy means to archive 
information. The use of the Web also fit nicely with other activities run by the YSP 
including a ‘'Computer Literacy Team" that tra\’clcd to schools to teach teachers and 
students about using the Web in the classroom. 

How would information be organized? 

A scries of "area codes" were created to specify the individual areas of science (anatomy 
= An; zoology = Zo). These codes coirespondcd with areas in which scientists could 
answer questions, thus providing a simple means of generating a pool of scientists to 
answer a given question. The codes are also used by CGI scripts to present listings of 
specific categories or to limit a keyword search to one or more areas. 

Given that two individuals oversaw the workings of the site, we decided to do as much as 
we could using the Web; this included reviewing questions, sending material to scientists, 
and checking information in answers. A scries of Web-form CGI scripts were written in 
Perl to carry out functions such as processing incoming questions and answers, 
generating listings of files on the fly, and providing a simple interface in which reviewed 
questions could be sent via e-mail to individual scientists. This collection of CGI scripts 
formed the basis of the Moderciijr package — the software developed to run the MAD 
Scientist Nctw'ork site. 
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In short, the final plan for receiving and processing information went as follows: 

1 . A Web form asked for the person’s name and grade level, the primary area of science 
represented by the question, the user’s question, and any further comments. Only the 
question and area of science were required. Students lacking e-mail addresses could 
still ask questions anonymously and receive an answer by referring back to the Web 
site. Questions submitted through this form were converted to HTML documents by a 
CGI script. During this process an ID was assigned to each question to assist in the 
tracking of files. 

2. After receipt, the question awaited review by a MadSci moderator. Moderators 
logged into a Wcb-CGI interface to review incoming questions and answers. The 
interface allov\'ed moderators to respond to questions directly (refer people to other 
Web sites, etc.) or to select a scientist to answer the question. Questions were sent by 
e-mail to the selected scientist. 

3. The scientist would then upload the answer through an online answer interface. As 
with questions, incoming answers were also formatted with HTML. 

4. The moderators would review the answer for content and accuracy. If the answer 
passed review it would be posted to the Cuircni Queue on the Web site: a carbon 
copy of the answer was sent to the person who asked the question if an address was 
provided. 

5. Additional CGI scripts provided indc.xing and search functions so that any visitor to 
the site could access accumulated question-answer information in the archive. 

Once the CGI interfaces had been adequately tested, the site officially went online. Fliers 
sent to the St. Louis public schools announced the service, as did presentations made by 
the YSP's Computer Literacy Team. Nonetheless, within a month, more than 99^/7 of all 
questions came from outside of the St. Louis metropolitan area. In fact, roughly 209; 
came from overseas! A number of recommendations were made by the initial users of the 
MAD Scientist Network. Our ability to respond to and incorporate these suggestions 
allowed our site to develop into the service it has become. 

Firstly, we received many requests from scicmisls at other institutions who were 
interested in answering questions. As volunteer sign up and question answering took 
place online, it took minimal effort to adapt our interfaces for use by scientists from 
around the world. 
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Secondly, the "other” category, originally designed to hold questions in biology and 
chemistry not covered by the defined areas, rapidly filled with questions concerning 
astronomy, physics, eailh sciences, and other topics. Rather than turn these questions 
away, we expanded our recruitment of experts by posting messages to Usenet science 
groups, as well as local departments in St. Louis. 

Over the next few years steady improvements in the Moderawr interface created 
necessary functions. Some of these functions included the automatic tracking and 
’‘recycling” of unanswered questions and the keeping of "expert statistics” (i.e., 
information on experts) such as number of questions answered, outstanding questions, 
deferred questions, etc. to assist the moderators when selecting a scientist to receive a 
question. 

The Web-based interface lies at the heart of the site. It distributes the review of hundreds 
of questions and answers among many individuals. It has permitted the expansion of our 
site from one that started with 40 experts and two moderators to one that has more than 
600 scientists and 20 moderators. This expansion has been necessary to keep astride of 
exponential increases in traffic. The site averages over 50,000 visitors a month and 
receives one to two thousand questions during this same period. Current efforts are 
directed towards means of checking incoming questions against existing answers in the 
aiv'hivc and structured organization of "exemplary” answers in the archive into FAQs 
covering specific areas of interest. 



Lynn Bry, M.D./Ph.D., lynn@}}iadsciM'iistl.eclii, is administrator of the MAD Scientist 
Network, and a pathology resident at Brigham & Women’s Hospital, Harvard University 
Medical School. 
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/ Getting Underway: The Virtual Reference Desk 
\ at the National Museum of American Art 



by Joan Slahl 



Introduction 

In 1993, when the National Museum of American An launched its e-mail reference 
service on America Online (AOL), '‘digital reference" was not a term in wide usage. Staff 
alternately and awkwardly refeired to the outreach initiative as “online reference,” 

“virtual reference,” and “e-mail reference" (the most accurate, but not the catchiest 
appellation). The service was considered so unlike traditional reference that the 
museum’s library staff wanted no part of it; in fact, I embarked on the project alone with 
support from the museum’s administration. Today, the environment has vastly changed. 
Many librarians around the country have taken their cue from the growth of AskA 
services and are offering digital reference service in conjunction with traditional 
reference. But the waters are still largely uncharted, and we can learn from one another as 
we initiate and develop our programs. With the clarity that hindsight affords, I can share 
some brief history of the National Museum of American Ail’s service and offer 
suggestions for planning similar services. 

History 

The Smithsonian Institution was approached by AOL to become one of its small but 
growing number of partners. The arrangement seemed a win-win situation. The 
Smithsonian Institution was free to provide whatever information we wanted, and our 
museums were the first represented on AOL. The Smithsonian Institution is a large 
museum complex, and much of its operations are decentralized. Accordingly, the 
participation in AOL was entirely voluntary for each museum. 

The National Museum of American Art was the first museum to participate; this seemed 
wholly appropriate, because the museum was at the forefront of the Smithsonian 
Institution’s experimentation with new technology. Although the museum has a national 
mission, it has nevertheless long operated as a local Washington museum. The director 
and staff were interested in exploring ways to reach an audience outside our region. Our 
decision to start an area on AOL should not be inteipreted as a way to exclude any 
portion of the electronic community; quite the contrary, it has always been the museum’s 
intention to reach the widest audience possible. We began with AOL for a few reasons: 
AOL was local, the arrangement required no initial outlay of funds for either staff or 
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cquipmcni, and the museum had not ycl de\'clopcd its own Web site. In shoii, ihe 
opponunily presented itself. The interactiv e features at the museum site on AOL included 
online chats, message boards, entcnaining contests, suggestion boxes, and a digital 
reference service. 



In the spring of 1997, the digital reference service was made available at the museum's 
Web site independent of the AOL site {lutp://iu}iaa-)ytlcr,si.edu/rcferc}}ccdcsk/). This 
allowed us to extend the serx'ice to a larger audience. More recently, the museum 
disbanded as site on AOL, while retaining a link from AOL to the museunVs service 
through AOLT Reference and Learn channel. 

.Mission 

The serv ice began quite simply. 1 managed the service by my.self and incorporated the 
additional duties into my responsibilities as coordinator of the muscunTs Image 
Collection, wiiich included overseeing our rather sizable slide and photo archive. I was 
happy to undertake the experiment because it provided me with the opportunity to keep 
abreast of emertzina electronic resources and technologies. At the time, I knew’ of no 
Other such service with a national constituency, .and 1 wondered w’hy librarians had 
quickly embraced document delivery and electronic reserves but were reluctant to 
advertise coiTCsponding electronic reference services. My aims or ground rules were 
uncomplicated and few: 

1. Increase and diffuse art inforinalit)n to a national audience. New technologies 
w'cre exploding, and the Internet was full of vast and competing resources vying for 
users' lime and money. Undoubtedly, the National Museum of American Art was 
better able than most other voices to provide information on American ail. 

2. Provide answers whenever possible via e-mail. While I referred patrons to u.scful 
resources, I also wanted to pnwidc answers whenever possible. Art information is so 
difficult to access in many public libraries that have had budget cuts. The service 
gives patrons brief, useful information that is conveniently obtained and is 
communicated in a personalized manner. In most cases the rcspon.se is a jumping-(^ff 
point for further research on the part of the pati\)n. 

L Use institution resources to the fuKesl. In addition to using the published rcsouiccs 
available in the museum's i)ulslanUing library rcsiuirccs, 1 incoipovatcd information 
from unique nuiscum-dcv eloped resoarccs including the Inventories of American 
Painting and Sculpture and curatorial files. 
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4. Recruit new museum members. The service could have a cost benefit lo the 
museum in terms of increased membership. 



Usage 



At the Sturt of the service, I received an a\'crage of 60 questions pct month. That number 
rose quickly, peaking at 537 questions in February 1998; the average number of questions 
per month is now 350-450. Use is heaviest at peak times during the school year 
(September to November and February to April) with decreases during holidays and 
summer months. Users come from all 50 states as well as Latin and South America, 
Europe, Africa, and Asia. They range from elementary students to professionals in non- 
art fields to artists, those interested in the arts, graduate students in art history, and ail 
professionals in museums and universities. The questions are the same as those that cross 
any reference desk but with a greater number of queries requesting electronic resources 
that can be remotely accessed. A small percentage ol‘ question.s arc “repeat questions": 
this is not likely the ease in other disciplines. 

Lessons Learned (Flanniing for a New Service) 

Nearly five years after the muscumN reference service began, I can better assess what 
docs and docs not work and identify the essential ingredients for the implementation of a 
successful program. 1 ha\'c retrofitted the initiative of the National Museum of American 
Art's Reference Desk to work bolter for both staff and virtual visitors. As you 
conicmpiale the initiation of a similar service in your institution, consider the following 
suggestions presented in the form of questions: 

1. Who \s your audience and what is the scope of your service? 

In other words, juslily the purpose of the service. Is there a need for it? If yes. it is 
important lo understand what information you can and cannot proN'idc. Inevitably, you 
will receive quesiion.s "out of scope"; though they may be easy to answer, you cannot be 
all things to all users. Beyond that, you need lo determine a reasonable turn around lime 
for questions received. 

The need for the Rel’crencc Desk at the National Museum of American Art is due to the 
nature of art books and their audiences. Ail resources arc cvpcnsivc and serve a smaller 
audience than do resources in the sciences and social seienccs. Consequently, most public 
libraries and many college and unixersily libraries have limited coMections, and virtual 
visitors arc not always well-scrxed when referred to such resources. Our statistics are 
ongoing testimony lo the need for the Reference Desk, hut the quantity of questions 
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received is also the reason that we do not answer questions that fall outside the scope of 
the service. For those questions on Afncan an or medieval art. for example, I send a 
polite reply such as: 

received your request on . You have wandered astray. The online 

reference service of the National Museum of American Art is dedicated to 
answering questions about American visual art and artists. Questions on other 
subjects are outside the scope of the service.” 

In planning your service, it is helpful for both you and the patron to estimate the turn 
around time. To that end, the Reference Desk Web form has a drop-down window that 
the patron can mark, indicating that he or she needs the information in one week, two 
weeks, one month, or any time. We strive to respond within five working days but advise 
patrons that the response time may vary based upon the traffic and the difficulty of the 
question. 

2. Does the digital reference service reflect a broader department or institutional 
mission? 

If the service reflects a broader mission, it is likely that you will be more successful in 
garnering both financial and technical suppon for the operation. The Smithsonian 
Institution’s mission is broadly stated — '‘to increase and diffuse knowledge.” Digital 
reference services are easily reflected in this statement. My effons are focused on 
creating linkages between the Reference Desk and the objectives of the museum. For 
instance, the service is limited to the subject of the museum’s collection (American art), 
and the Reference Desk serves the same audience that the museum strives to capture 
(national and international). Over time, the museum’s administration has embraced the 
Reference Desk because it sees that the service carries out many of the museum’s 
objectives. 

3. Have you identified staffing needs and a reference model? 

A one-person service has some appeal because you can oversee all aspects of the 
program. But is it practical? If not, how can you distribute queries and to vvho.m? Is there 
a model of reference service that correlates with the service you want to deliver? 

Over time, I ha\c come to understand that a one-person service is impractical and 
difficult to maintain. But 1 am still trying to determine an acceptable and fundable way to 
distribute questions both inside and outside the institution. 
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4* What is your funding plan? 

Digital reference does not pay for itself unless it is a fee-based service, if your service is 
not fee-based, how will you pay salaries and purchase equipment? Can you work with a 
development officer, contract out for such services, or undertake development yourself 
along with other responsibilities? 

I have no magic answers. Based upon my experiences, as the service is more heavily 
used, funding issues will surface (due to increased staff, technology upgrades, etc.). At 
the National Museum of American Art, we are undertaking fund-raising initiatives to 
supplement the museum’s support. 

5. What is your marketing plan? 

If you are making the effort to develop a digital reference service, do not overlook the 
need to advertise it. Consider appropriate places to advertise, based upon the profiles of 
your anticipated audience, I have often been asked why the Reference Desk retains a link 
from AOL to the museum’s Reference Desk on the Web. From a marketing standpoint, 
the reason is obvious: A.OL is the country’s largest Internet provider, and many of its 
subscribers do not have other modes of access to the Web. 

Marketing considerations will cover a range of possibilities including 

• Placement of the service on your Web site (top level or buried several levels down) 

• Advertising in institutional new'sletters or publications 

• Links on other Web sites that share subject interests 

• Fliers and handouts. 

The possibilities are limited only by your time and creativity. 

6. What equipment and technical support are needed to operate the service, and are 
they accessible to you? 

Digital reference is a technology-reliant service. It is critical that you have hardware and 
software to support the operational activities that are components of the service. The 
service you develop may require software customization and will certainly require 
technical support in the face of system problems. Is such support available in your 
institution, or do you need to coniracl for it? 
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Technical support is an issue for the Reference Desk; the service has outgrown its simple 
organization. It now requires support from the museum’s Office of Information 
Technology, which is already pulled in several directions. With better planning, 1 might 
have been able to better prepare for the technical challenges the Reference Desk is now 
facing. 

7. Who will train staff and what training is necessary? 

If the questions are to be distributed, or if the service continues after you leave your 
present position, it is useful to identify staff training needs. A procedures manual may be 
time-consuming to create but will be well-utilized as your service grows. Additionally, 
you may want to create 

• Boileiplate statements that cover repeat questions and situations (In the case of the 
Reference Desk, for example, I have statements about locating an appraiser and 
searching auction records.) 

• Lists of useful Web sites 

• Sample responses (to illustrate the opening, body, and closing of the response). 

Conclusion 

Digital reference makes sense. Information overload is a fact of life, and despite, or 
maybe because, so much information is accessible electronically, the average user often 
needs a human interface (a libraiian or information specialist) to assist in navigating his 
or her way to appropriate resources. If you start a digital reference service, be assured 
users will come. It is easy to be overwhelmed by success and in shon order find your.sclf 
unable to deliver the service you advcilisc due to the quantity of questions received. To 
avoid that situation, be realistic and plan. 



Joan SVdhlJsialiKa nniaa.si.eclii. is the Coordinator of Image Collections for the National 
Museum of American Art, the Smithsonian Institution. 
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Module 2 Summary 

This module provided background and techniques Ibr creating a plan Tor new AskA services based on 

organization goals, a generalized service structure, and the experiences of existing AskA ^crvices, 

• The AskA Plan is a document that outlines the potential components of a particular AskA service. 

1 . Service goals should reflect goafs of supporting orgunization(s) and should consider issues 
regarding potential audience, benefits of service to stakeholders, staff, users, etc. 

2. Organizations can benel'it from supporting digital reference service in several ways including 
providing a method for collecting feedback from u.sers and stakeholders; promoting national 
educational initiatives: and impacting education. 

• All AskA services can be described in terms of a conceptual framework with three main components: 
methods for obtaining information cm the environment (detectors); proces.ses for building and 
providing .service (rules), including resources for carrying out rules; and ultimate services provided 
(effectors). 

• Organizations can create and implement their oun AskA j^lan based on the general structure. 

• Examples and experiences of exemplary .AskA services can offer guidance and ideas in planning new 
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Training Ask A Service Staff 

Module Profile 



Informing 

Planning 




Training 


Prototyping 

Contributing 

Evaluating 





This module is designed to assist organizations in planning, delivering, implementing, 
and managing training programs for AskA service staff and information specialists. 
Topics covered include the imponance of training staff of AskA services and methods for 
creating a training plan, producing the training materials and delivery tools, and 
implementing and managing the overall training program. This module is divided into 
three main sections to accommodate organizations at different stages in training 
development: creating the training plan, producing the training program, and 
implementing and managing the training program. The three sections are preceded by an 
introduction to AskA training. At the conclusion of this module, you will be able to 
achieve the following goal: 



Goal Plan, produce, implement, and manage training programs for 

service staff and information specialists. 

Prerequisites Before participating in this module, you should be able to 

1 . Collect information on the general digital reference field and 
existing AskA services (sec Module 1: Informing). 

2. Determine the best way to build and maintain digital reference 
service within the organization (see Module 2: Planning), 
including human resources, policies, and procedures. 

Objectives At the completion of this module, you will be able to achieve the 

following objectives: 

1 . Understand the importance and puipose of planning an 
effective training program, 

2 . Identify staff members vvithin the service who require training. 

3. State three general phases involved in creating a training 
program for AskA service staff. 

4. Create a plan for an AskA service's training program, 

3. Produce the training program by creating materials, preparing 
delivery tools, and piUu testing materials and activities. 

6. Implement training program according to planned schedule, 

7. Manage training program by ongoing evaluation and revision. 
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3.1 Introduction to AskA Service Training 

In a digital reference setting, iraining refers to the planned preparation of individuals 
involved in ihc creation, maintenance, and operations of a particular AskA service. As 
e.xplaincd in the Introduction, this Starter Kit is intended as a set of self-instructional 
modules for AskA service administrators — those involved in the creation and 
maintenance of an AskA service. In this way, the Staiier Kit as a whole serves as the 
main training component for AskA service administrators. 

This module focuses on the steps necessary for planning training programs for those 
involved in AskA services on an operational level: information specialists (those 
responding to user questions) and support staff (those involved in operations other than or 
in addition to question answering). The steps presented in this module are based on the 
process of instructional design — a systematic approach to planning learner-focused 
instruction (Dick & Carey, 1996; Gustafson & Branch, 1997) — and experiences of 
existing AskA services. 

Importance of Training 

Effcctix c training programs for AskA services are important for many reasons. The 
digital reference field involves innovative uses of technology, new aspects of reference 
and educational service (i.e., virtual vs. facc-to-facc), and other situations that demand a 
special set of skills. In addition, each AskA service has its own distinct policies and 
procedures that arc to be followed by all staff and information specialists. Finally, AskA 
services rely on well-packaged, reproducible training programs and materials in order to 
accommodate the constant increase in numbers of information specialists and staff 
members as the service grows. 



Who Is Trained? 

The main audiences for training in an AskA service arc information specialists and the 
staff who facilitate the w ork of information specialists and the overall service. (See 
Figure 3-1 for training functions by audience. Items in bold indicate areas targeted by 
individual Ask/V service training programs.) 
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Fij^ure 3-1: Training* Solutions for AskA Service Stuff 



DilTereni training programs arc required lo prepare caoli audience for its role as outlined 
below. 

Tra i niny, b ifornia I ion Specialists 

AskA services generally place their largest training clToils on the individuals who 
respond to user inquiries. The main reason for this emphasis is that information 
specialists have the most direct interaction with users. The success of a digital reference 
service highly depends upon the effective interactions between information specialists 
and service users. Also, information specialists often make up a greater number and a 
more geographically diverse group of people than those in other staff roles do; therefore, 
more attention to logistical training needs is required. 

Depending on a service's situation and available resources, training information 
specialists for digital reference service frcqucntlv' consists of familiarizing information 
specialists with service policies and question answering procedures, sometimes allow ing 
oppoilunities to practice responding to inquiries. Training ma\ also include instructing 
information specialists to answer reference questions in general and lo communicate 
cfrcclivcly using the Internet. 

Implementation of training for information specialists varies from sciwicc lo scr\ ice. 
Some offer formal programs via distance or facc-to-facc, while others simpl\- provide 
written resources (c.g., manuals) for information specialists lo revicv\ before and during 
participation. See “Select Delivery .Methods'' and “Sample AskA T'raining Mi^dcls" for 
e.\amplcs of various types wf training programs. 
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Training Support Staff 

Individuals who conduct day-to-day operations of a service may require training in 
service policy and procedure as well as use of resources and technology necessary for 
accomplishing given tasks (e.g., question routing, monitoring information specialists' 
responses, Web site development and maintenance, etc.). 

Suppon staff may be further divided into different job positions and tasks (e.g., 
mentors./leam leaders/administrators, technical support, etc.), each requiring slightly 
different training programs. This module highlights support staff as one group, although 
services will make training decisions based on individual situations. 

Training programs for staff may include a written manual with service policies and 
procedures and specific job roles; some may involve participation in the question-answer 
training designed for information specialists as a way of becoming familiar with those 
operations. Training programs for support staff are often less formal than those for 
information specialists. 

Who Flans, Produces, Delivers, and Manages Training Programs? 

As will be discussed further in this module, the overall AskA training process can be 
broken into distinct phases requiring the expertise and efforts of different people. The 
planning of overall training programs and the development of activities and materials is 
often carried out by service administrators; however, these tasks can also be performed in 
conjunction wdth outside instructional design or training consultants. The implementation 
of training programs — which can involve delivering face-to-face sessions, electronically 
distributing materials, providing feedback on practice activities, etc. — can be conducted 
by service administrators, designated staff members, or information specialists who have 
demonstrated quality work and responsibility. 



i I 



78 



Training 



Module 3 



Phases in Building an AskA Service Training Program 

There are a lot of issues to consider when creating any kind of training program from 
start to finish. This module breaks down the overall process into three separate phases 
based on principles of instructional design. AskA service administrators can follow these 
steps to create the most effective and efficient training programs for their AskA services: 

L Creating the Training Plan 

2. Producing the Training Program 

3. Implementing and Managing the Training Program 

The three phases are fuither broken down by specific activities and questions conducted 
within each phase: 

1. Creating the Training Plan (see Module 3.2) 

• Establish training goals — What should information specialists and support 
staff be able to perform as a result of training? 

• Select and organize training components — How should information be 
presented and training activities be administered in order to get the most out 
of the training program? 

• Select delivery methods — What types of media and tdecommunicalions tools 
should be used to deliver information and conduct training activities? 

• Create a learner-assessment plan — What criteria will be used to determine 
whether or not trainees are prepared to perform required tasks following the 
training program? 

2. Producing the Training Program (see Module 3.3) 

• Create training materials — What are the most efficient and effective ways to 
develop and organize planned materials and activities? 

• Prepare delivery tools — What is involved in securing and preparing tools and 
equipment to implement program? 

• Pilot test — How will the program be tested in order to determine if training 
materials and activities will be successful? 
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3. Implcmcnling and Managing the Training Program (sec Module 3.4) 

• Schedule trainine events — Whal arc the best ways for managing the overall 
training program crfcclivcly and cfTicicnlly? 

• Provide feedback to trainees and stalT — Whal arc the most elTicieni and 
effective ways to communicate feedback to help trainees and staff improve 
performance and slay motivated? 

• Ex'aluate and revise the program — Whal efforts will be taken to e\*alLiatc the 
training program on an ongoing basis (e.g., instruments, analysis, revision)? 

Each of the three phases w ill be discussed in detail in this module. The p'anning phase 
section contains the most information and examples since the results of planning set the 
stage for the following phases. The sections on the producing and the implementing and 
managing phases illustrate ways to put the training plan into action. Specific 
recommendations and examples from training programs will be presented in each section 
to help administrators plan and implement the most appropriate training programs for 
their AskA ser\ ices. Although the phases are discussed separately and in a distinct order, 
it is expected that one will travel between stages throughout the process as appropriate. 



Module 3.1 Summary 

Thisseciion provided an imroduciion to training of AskA scrviuc staff. 

• Training programs for AskA sors ices help staff prepare for special situations involved in digiUil 
reference and understand and carry oiil scr\ ice pedicy and puicedures. 

• Training programs Tor .AskA services sIkhiUI be reproducible and accessible in, order (o accommodate 
(lie constant need for training ncu slat!'. 

• The main audiences for training in an A.'.kA .service arc information specialists and support slatf \sln> 
facilitate the work of information specialists. (This latter group may be further broken down depending 
on specific service roles.) 

• The planning t>f training pmgrams may be earned out by AskA service administrators, perhaps in 
conjunction with instructional design consultants; production, delivery, and management of training 
programs can be carried out by many different people including service administratt^rs, designated staff 
members, or information specialists svho have successfully completed training activities. 

• The three mam phases for building a training program are creating the training })Ian, producing the 
training program, and implementing and managing the training program, 'fhe phases are often 
conducted in a nonlinear manner. 
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3.2 Creating the Training Plan 

This section describes the first phase of the ox'erall process for creating a training 
program for AskA service staff: planning digital reference training programs for 
information specialists and support staff. The two main training audiences v\’ill be treated 
as separate entities although sonic issues may be the same for both. Also keep in mind 
that support staff will include one or more distinct positions depending on the service and 
thus may require more than one type of training program. 



Establish Training tloals 



Before thinking about glossy manuals or Internet-based inlcracii\‘c training sessions, it is 
impor int to focus on what the training audience must learn in order to cairy out 
responsibilities. This step should occur at the beginning of the planning process hecausc 
all other decisions will be based upon the information collected at this time, including 
cuiTcni skill level of information specialists and support staff and the tasks they will he 
expected to perform. 

This step in\’o!\cs four major activities as represented by the fcdlowing questions: 

T What tasks will information specialists and support staff need to ncrl'orm in 
(a'dcr to carry out responsibilities? 

2. What skills arc required of information specialists and support staff in order to 
perform identified tasks? 

3. To u hat extent arc information sjiccialists and support staff prepared to perform 
required tasks? 

4. What will the iF'aining program attempt to accomplish in terms of preparing 
information specialists and support staff to perform required tasks? 

The above questions can be answered by survey, observation, and analysis as described 
funher. 

/. U7/C// tasks will information specialists a/ul su/'fport staff need to perf)nn i)\ order to 
carry (mt rcspoi\sihilities? 

The tasks to be performed by information sjiccialisls and support staff will be determined 
mostly from job descriptions already identified (according to the original AskA plan), but 
a more formal analysis may be taken to map out skills on a detailed level. Each audience 
(within the categories of information specialists and support staff) sheudd be analyzed 
separately to find distinct tasks, although it is common for some to overlap. 
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For example, the primary task of an information specialist may be to answer questions 
about the subject of specialty (science, math, etc.) using expcn know'ledge. But what 
other steps are involved in answering questions? 

• Are questions sent to the information specialist's personal e-mail account from a 
central account, or does the information specialist need to retrieve the inquiry 
message from a central location (e.g., Web-based system)? 

• Does the response message need to include certain components (e.g., reference to 
resources, greeting to user, statement from service)? 

• Arc there any specific software or Internet functions that must be performed (e.g., 
Web .searching, use of e-mail package)? 

• Whal should information specialists do with the response after it is written (e.g., send 
it directly to the user, post it on a Web site, etc.)? 

• What actions should information specialists take when a problem occurs (e.g., 
incorrect user address, unable to complete response within required time limit)? 

If not previously answered, these questions should be examined now. One way to ensure 
that all steps are considered is to complete the task yourself or observe someone else 
completing the task. Make note of every step taken to get from the beginning to the end. 
Afterward, cicate a chart or list of all of the steps. 

Figure 3-2 shows a sample task analysis in chail form adapted from an existing AskA 
service. 
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2. WImi skills arc rcijiiircd of infonnafion spccialisis and snppori staff in order to 
perform identified tasks'/ 

Al'lcrihc specific tasks arc examined, it is possible to idcniily the skills necessary to 
ccMiipIclc each task (sec Table 3-1). Some skills will be considered prerequisites, 
meaning they will not he cos'crcd in the formal training program; information specialists 
or support staff are expected to have these skills before participating in the service. Those 
not identified as prerequisites are areas that ought to be covered in the training program 
in some \N ay, either as major topics of instruction or supplemental information. (Specific 
methods for covering such areas will be addressed later.) 

F-^rerequisites will be determined mainly by survey of users (sec question 3 below) but 
also by scr\ ice requirements and limitations, Prciequisites must be stated upfront as 
requirements for participation. 

Aside from required skills, information specialists and support staff arc expected to have 
certain technical capabilities in order to participate in the service. This may include 
regular access to a personal e-mail account, Web browser, or word processing program, 
and the ability to use more than one application at a time. It is important lo distinguish 
between necessary capabilities and those capabilities that would facilitate completion of 
tasks. (For instance, in some cases it ma\' be helpful to have access to both e-mail and a 
Web browser at the same lime for question answ ering, although it may not be required.) 
'These capabilities should also be sttited as requirements for participation. 

0 . To what extent are inforniation specialists and support staff prepared to perform 
reepdred tasks '/ 

Once the specific tasks are identified, it is necessary to determine how well the training 
iiudience is currently prepared to perform them. According to the specific ttisks and 
.seiA'ice pcdicies and procedures, the following information may be sought: 

What (and lo what extent) do potential information specialists and support staff iilrc\id\‘ 
know about 

• 'The subject or tojne in which the service specializes? 

• 'The user audience? 

• Tsing necessary Internet tools and functions: Web browser, e-mail, Web design? 

• I'sing specific software applications or functions? 
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• Identifying and locating information resources (and information within resources) to 
support responses? 

• (.'ommunicating effectively and appropriately using the Internet? 

• Providing feedback on information specialists’ responses? (for support staff). 

Answers to some of these questions will clarify any assumptions made about what the 
audience cuiTcntly knows before training (prerequisites) and consequently provides 
information on what should be included in training. 

This type of information can be gathered through standard survey mechanisms such as 
written questionnaires, interviews, focus groups, etc. with potential information 
I specialists and support staff. Perhaps the easiest, quickest method of collecting this 

information is a questionnaire distributed via e-mail or on a Web form. See Figure 3-3, 

I ‘‘Pre-Training Questionnaire for New Experts.’' 

I People to target include those who have already c.xpresscd interest in participating in the 

service and people who you would expect to have similar background knowledge to 
I potential service staff (c.g., high school English teachers for a lilcraiurc-rclaicd AskA 

service). jMcthods for recruiting individuals to respond to survey questions may include 
I posting requests on a Web site or on e-mail discussion lists. 

I 

I 

I 

I 

I 

I 
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Figure 3-3 Pre-Training Questionnaire for New Experts 

Please eheck the most appropriate answer so we can make our training program right for you! 

1. How many years have you been working in this particular subject area? 

I- 5 

6-10 

II- 15 

16 or more 



2. How often do you use e-mail? 
7 times a week or more 
4 to 6 times a week 
I 10 3 times a week 
Less than once a week 
Never 



3. What do you use e-mail for? 

Work 

Lisiservs (Electronic discussion groups) 

Social correspondence 

Other: 

4. Where is your Internet connection? (Check all that apply) 

Home 

Work 

Other: 

5. What type of WWW access do you have? 

Text only (Lynx) 

Text and images 

6. What informalion resources are available to you? 

Internet 

CDs 

Online databases 

Print resources 

Other: 

7. What skills would you like to acquire (or sharpen) in preparing to carry out your 
responsibilities as an "expert"? 



8. What information would be helpful to you to know in preparing to carry out your 
responsibilities as an "expert"? 
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Table 3-1 contains a list of tasks with corresponding skills. The tasks included arc 
adapted from a manual for information specialists of a subject-oriented AskA service 
(MAD Scientist Network, 1997a). Prerequisite skills are indicated by a check mark in the 
prerequisite column. Those skills that are expected to be covered in training are in bold 
type. Skills vvill be later translated into objectives and then met with specific training 
materials and activities. 



Table 3*1: Tasks and Corresponding Skills for Question-Answer Procedure 



Prerequisite? 



1. Open e-mail message. 



2 . Choose “respond” command 
or open ncKv file :n word 
processing program. 



3. Greet user by name. 



4. Provide factual answer. 



5. Provide references to 
additional resources on topic. 



6. Sign name at end of message. 



7. Resolve problems during 
question- answer process. 



8. Paste response intoemline 
messaoe form 



send e-mail response lo Web site 
administrator. 



1. Proficiency with personal c-mail program 



2 . Proficiency with personal e-mail program or 
proficiency with word processing program 



3. Knowledge of service question-answer policy 
and procedure 



4a, Subject expertise 



4b. Ability to communicate effectively via 
Internet 



4c. Familiarity with user audience (K-12 
students) 



5a. Knowledge of overall reference process 



5b. Use ot periodical indexes 



5c. Use of library or collection catalog 



5d. Web-searching skills 



5e. Knowledge of HTML (to include links) 



6. Knowledge of service question-answ er policy 
and procedure 



7. Knowledge of service question-answer policy 
and procedure 



8a. Proficiency with Web browser and word 
processing functions for copy and paste 
8b. Ability to open more tfian one application at 
once (optional) 

8c. Proficiency with personal e-mail program 
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Work Sheet 3-1: Task and Skill Table 



Task 



Skill 



Prerequisite? 








The case study "Elccimnic Reference Service Issues and Necessary Skills" describes 
necessary skills for particular situations involved in the question-answer process from the 
perspective of one public library electronic reference service. 

4. Whai will ihc frahiifi^ program auempt lo accomplish in terms of preparing 
information specialists and support staff to perfonn reepdred tasks'/ 

Alter identifying what the training audience must know and what the audience cuircnlly 
knows, you can determine where the training efforts should focus. Building on the 
analysis represented in Table 3-K it is possible to create a goal statement lo define what 
the training program will try to accomplish (in terms of trainee performance). From this 
statement, you can later create specific objectives to guide the content of your training 
programs. Documentation of this type is not only helpful in guiding pre-training activities 
but also aids in justifying training efforts lo organizations supporting the scr ec. 

Below arc examples of training goals for information specialists and support staff: 

Information Specialists Goal 

Information specialists wili respond to user inquiries using appropriate information 
resources and individual subject expertise. 

Support Staff Goal 

Service moderators will support the work of information specialists by reviewing and 
organizing incoming questions; managing, reviewing, and revising information 
specialists’ responses; and providing feedback to information specialists regarding 
responses. 

Select and Organize Training Components 

This stage offers opportunities lo map out the infoiTnaiion, ideas, and activities to include 
in a traininiz prosram. Also at this time, many decisions arc made rcaardina how trainine 
materials will be developed and organized. Some questions answered during this stage 
include the following: 

1. What skills must the training audience have in order to achieve the ultimate goal? 

2. How should necessary information and ideas be communicated lo ilic training 
audience? 
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/. Wliaf skills must the training audience have in order to achieve the ultimate goal? 

In order lo achieve the previously identified goals, information specialists and support 
staff members will demonstrate a variety of other skills identified during the tusk 
analysis. These skills are planned out in the form of objectives. Some objectives refer to 
uliimialc goals while others indicate steps necessary to reach those goals. Objectives not 
only set out what the information specialists and support staff should do after training, 
but will help guide the administrator or instructional designer in presenting content. 

Below are some sample objectives for information specialists who refer users to 
information resources: 

1. Prepare a response lo a user inquiry by opening message in personal e-mail account, 
selecting “reply” and “include original message” options, placing user address in 
reply line and service address in the cc line, and greeting user by name. 

2. Research user’s question by searching the Internet and other print and electronic 
sources. 

3. Respond to user’s question by typing or electronically transferring information from 
the original source to the reply message, 

4. Conclude response by refeiring user to additional sources, signing name, and sending 
message. 

3. Resolve problematic situations during response process according to service 
guidelines. 

Information specialists who arc required to use their own subject knowledge rather than 
information sources may have a different set of objectives including these: 

1. Acknowledge user’s previous effoils in answering question. 

2. State concepts in age-appropriate language. 

3. Allow the user oppoilunities to answer similar questions on his or her own by 
providing sample questions, suggesting additional activities, or both.'' 



' This example wab adapted from Ask Dr. Math's Guulc to Writing, Responses. 1998 
http: //forum. swarthfHorc. cdu/dr.nuuh/^uidc/. 
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Support staff may be required to perform a variety of tasks such as the following: 

1 . Use centra! system to review and route incoming questions to appropriate information 
specialists. 

2. Review information specialists’ rcspon.scs according to set of criteria outlined in 
service guidcUnes for responding to inquiries. 

3. Provide feedback to information specialists using appropriate language and tone. 

4. Locate Internet and other sources to include as references (supplcmcnta’ information) 
in previously written responses. 

Once the goals and objectives arc set, it is easier to sec where to concentrate the training 
efforts. The following sections will illustrate how training components arc selected with 
help from the objectives. 

2. How slioiilil necessary in/onfialiofi ciiui ideas he eofwunnieaied lo the trainifi{^ 
audience? 

For each objective identified, choose the most appropriate treatment: the content and the 
activity or materials to communicate the content. For instance, if an objective states that 
information specialists are expected to respond to user inquiries by following certain 
procedures, an appropriate treatment may be to j^rovidc opportunities for the trainee to 
compose practice responses. 

Objectives that imply general skills — such as Internet searching, info nation skills, 
electronic communication skills, etc. — arc addressed in a document created by the Virtual 
Reference Desk project. Giddelifies for hifonnation Specialists of K-12 Diy^ital Reference 
.SVn’/c‘c.v (Kasowitz, 1998) can be used in conjunction with training pi'ovidcd by 
individual services. It offers general tips and suggestions for providing information to K- 
12 community members o\ ir the Internet using an information problem-solving approach 
(a method for addressing questions by applying critical-thinking skills and assessing 
information resources). 

Choose and Organize Content. The first step in selecting content should he to analyze 
the objectives established from service requirements and audience characteristics. For 
instance, if an objective states that information specialists should refer users to Web sites 
as part of the response message, it may he necessary to offer some guidance on how to 
search for and evaluate Web sites (this is assuming that a certain number of trainees arc 
not already proficient in this area). See Table 3-2 for an example of tiic evolution of 
objectives into content and then possible treatments. 
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fl is also helpllil lo consider common topic areas highlighted in AskA service training 
programs. Prom an initial analysis of training materials of exemplary AskA services, 
several areas have been identified (keep in mind that many areas come directly IVom 
individual service p(')licies and procedures). (See Table 3-2 for these commt)n areas and 
related contents included in services’ training materials.) 
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Tahlf 3-2: C’oimnoii Content Aicns in AskA T raininj’ MateriuKs 



All 



TypcbfStaJf 



Areqs’ - ;' ' • Content ^amplQs:- A - i 


(leneral ni'oniuiiion and hackground 
about seiAiee 


• ITineiions am! individuals invoUed 

• ( ‘oniact mloi maimn 

• Sponsoi mg or supjioi ling organvaiious 

• Seruee mission, goals, aiui iihjeeloes 

• 1 lisioi \ oT ser\ lee 


Desei iption of |oh role 


• Kespoiisihilities/eNpeeiaiKuis 

• Required skills 

• Required lechmeal capahihlies 



(Jiicsiuni-aM>^\ci pnH’cdmv 



Support >uilT 



l^itk'odiirc.s lor nniimii ipieslioiis uiul 
e'. aliMimg icsponscs 



Pfohk‘in sol\ imi/T foublcshootimj 



I-Acmpkir\ ^c.^po^l^e^ 



Kole or use ol su|^jioitint: I'Csouiecs 
iins'Aomii: ijvicsiioiis 



Insii LieiioMs jor 

• Uccemneoi elauniiii: questions 

• UeM.Mu hinp cpiesiuMi (releience process > 

• Writing responses 

0 Recjuired eoinponetiis 

V (leneiiil iipj'-ioaeh ami tone 

(■ Suggested lomiiit 

• Semlmg i espouse 

e Tune limit 

0 Wlial liappeiis to response 

• ‘l eehnolo!!) and intei laces used in q ,i 



• (Yiierui lor e\alii:iiMU' IS lespunses 

• Suggestions IiM proshliMg teedlKick 

• Technology used 



'nsimetioiis lor handling situations such as: 

• IS eau't answer t|uestion w iihiii time limit 

• IS Ctin‘1 I'lnd answer 

• I 'ser (juestion isunele.u, mappmeu lale. etc 



Iiiehide esamples 

• Within ie\t to illustrate points 

• As separate section or ivsoiiive 



• Sitilc possible snou es 

'/ Internal tindcscs. aiehive. l-AQsi 

V l Aleinal st>uices t W'eh sites, icvtsi 

• rro\ ide support lot using liie Weh 

0 I \sing search engines 

V 1 Aalualinu Weh sites 

• Inslruelions tor incorporating citations oi 
siuirecs w ilhin response 

V 1 inking to Weh pages 

0 Cploaduig Tiles 



BEST COPY AVAILABLE 
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Aiiolher possible urea to consider is information or directions for updating, delivering, 
and managing the training program. This ty()c of information in the form of a “trainer’s 
manual” or “instructor’s guide” would he useful for staff members involved in these 
functions, especially in situations where staff changes occur. 

(’hoose Materials and Activities. Once you’ve iderhfied the content to include in the 
training progrUi.T it’s time to choose a format to communicate the information. The two 
general categories for formats arc materials (c.g., handbooks, manuals, how-to lists) and 
activities (c.g., shadowing, practice questions with guided feedback, etc.), vS]')ceific media 
Tor presenting information and strategies used to cairy out activities will be discussed in 
the following section, “Select Delivery Methods.” 

Tlic bdlowing table (3-3) shows sample ohjcclivcs and their coircsponding Ircatmcnis in 
terms of content as well as possible materials or activities. The table is intended as a 
general guide to present common ohjccti\'cs and treatments u,sccl in AskA service training 
programs. Keep in mind that individual services will require treatments that address 
objectives based on special responsibilities and skills such as sg/ccific queslion-ansv^'cr 
procedures, software or Iniernel tools u.scd to operate the service, information resources 
specific to the organi/.alion or content area, etc. 



i i 1 ^ 
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Table 3-3; Possible Treatments Based on Specific Objectives 


Objective ‘ 


. ‘ Treatment ^ : r| 




Content 


Materials/Activities 


1 . Prepare a response to a user inquiry by 


• Service guidelines for 


• Policy and procedure 


opening user’s message in personal e- 


responding to user 


manual 


mail account, selecting “reply” and 


inquiries 


• Checklist of steps for 


“include original message” options. 


• Steps for completing 


completing response 


placing user address in reply line and 


response 


• Checklist of required 


service address in cc line, and greeting 


• Instructions for using 


response components 


user by name. 


technology necessary 


• [examples 




to perform response 


interspersed 




• List ('ll* required 


throughout manual 




response components 


• Practice questions and 




• lixv nples of 


feedback on practice 




successful responses 


responses 


2. Perform and promote information 


• Steps in general 


• Resource on general 


problem-solving process in responding 


information problem - 


digital reference skills 


to user questions. 


.solving process 


(Kasowiiz, 1998) 




• Lixcmplary responses 


• Section of manual 



wiili service-specific 
exemplary responses 
containing 

informalion problem- 
solving lips 

3. Create clear and effective message Criteria and examples lor 

using clear language and layout* using a creating clear and effective 
friendly and appropriate tone, and messages 

motivating users to learn. 



• Resource i)n general 
digital reference skills 
(Kasowilz, 1998) 

• Sect iim of manual 
wi i h sc r V i cc-spcci f ic 
examples of 
exemplary messages 
and poor quality 
messages (non- 
exam pics) 
















Work Sheet 3-2: Training Objectives and Solutions 



Objective . • ' Treatnieht 



Content Materials/Activitics 
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Select DelHery Methods 

During this sicp ii is imporiant lo decide how the ov erall training program should be 
implemented lo help the training audience achicv'c goals. This is the part that most people 
associate with a pailicular training program — the actual products and events. What will 
the whole training package consist of when it is complete? l-rom the situation in the 
prcv'ious table, we may envision the following two training scenarios (see "Sample AskA 
Training Models" for additional examples): 

Scenario One 

1. Participants arc directed to a Web-based policy and procedure manual witli 
checklists lo be printed out, exemplary responses, and reference to the VRD 
Guidelines document on the Web (Kasowitz, 199S). 

2. Administrator e-mails practice questions to each training participant once the 
participant has reviewed the materials. 

3. Training participants respond to practice questiems and return responses lo 
administrator for rev iew and feedback. 

Scenario Two 

1. Service coordinator distributes lo each participant in a facc-io-facc session print- 
based policy and procedure manual with checklists and pointers to VRD 
Guidelines document on the Web (Kasowiiz, 1998). 

2. Bach participant observes current infoimaiion specialists performing task of 
responding lo user question. 

3. Participants claim practice questions from Web site. 

4. Each participant returns practice response to trainer (a current information 
specialist) for review; feedback is delivered electronically and facc-to-facc. 

Clearly the main difference between the two .seenarios is that the first scenario delivers 
training er.: rely from a distance, while the second scenario involves some facc-to-l'ace 
interaction and direct observation. Since many ,services’ staff members arc widely 
distributed geographically and all AskA services use electronic communication in daily 
activities, most services employ some type of "distance training/"' For example, training 
manuals arc accessible on the Web, and practice activities inv'olve using the same e-mail 
or VVeh-based communications tools that w'ill be u.sed during service pailicipation. 



" I ‘or more inrormatuw) on di.siance education, sec Kerka i jVO()) and IdM'msi'owski t 
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Each service should decide for iiself how much of training lo deliver electronically and 
u hen, The following issues can be considered in making this decision: 

• Geographic location of training participants 

• Access of participants to Internet: speed of connection, type of Web browser, e-mail 
program 

• Resources of AskA service organization to develop and distribute electronic materials 

• Participants' skill with necessary tools upon entrance to the training program. 

The important thing to remember is that one training program can incorporate several 
diifcrent methods and tools (c.g., e-mail, Web page, group discussion) depending on the 
best way to communicate a certain idea, reinforce a concept, or improve and build a skill. 
This can include a combination of distance and face-lo-face components. 

Tabic 3-4 shows possible types of information and activities to be delivered as part of a 
training program and possible ways of communicating the information and carrying out 
activities. This is just a sample, but it includes Internet tools as well as print-based and 
facc-to-fucc interventions. Also keep in mind that several different types of tools and 
approaches may be used to deliver a given set of information or camy out an activity, but 
some tools and approaches arc more effective for a given set of trainees and situations. 
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Table 3-4; Possible Treatiociits and Delivery Tools Matrix 



Present 

Information 



Discussion . Vroctice Feed- 



Comments 




(MOO. IRC') 



Attractive presentation. 

Allows hyperlinks to 
basic and supplcmciiial 
inlormatuHi. 

Aiilhcniic pi aciicc 
possible tiimugli Web 
rornis. 

Allows access to Web- 
based service resources 
(archive, que.stion- 
answer distribution 
system). 



Limiied tdnnaiiinu 
options due to need to 
aecomnuKlate UNIX- 
based systems. 

Good (or asynchrtmous 
communicutum and 
auihcmic practice il’ 
question-answer 
prcK'ess is e-mail- 
ha sed. 



C'an allow sharing of 
different types of files, 
from simple ic.xi 
documents to files with 
graphics and ^'ideo. 

Inles can be viewed 
using downloadable 
reader stiftwarc such as 
Adobe Acrobat. 

biles must he 
compatible with 
participants* s v’ s t.e ms. 



Ciood lor synchronous 
communication with 
several participants. 

Sometimes diftieult to 
manage for large 
groups, 

Software must be 
eomjxmbic with 
par l ! e i pant s ’ sy s tc m s . 



'' Ji>h aids are supplemental resources intended to help the trainee come up li’i speed during training and to 
pro\ uppori tor performance on-the- job. 



t 
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•Pelivery . Present . . Discussion Practice Feed- Job Comments 

I Tools ' Information ^ back Aids'® 



Prim -based 
materials 



1 list nitior- led 
presentation 



Jnsiriietor-Ied \ 
ela.s.s 



One-on-one 

inentorini: 



C'omputer- 

based 

instiueiion 




Good for face-to-taee 
sessions. 

Doesn't alUnv praciiee 
in a Lit hemic settiim but 
can help sharpen some 
skills. 

I'or distance irainine, 
cc^py and mail costs 
may exceed costs lor 
Croat ing and 
distributing electronic 
resources. 



Good for introducing 
topics; must tolling up 
with review and 
practice and feedback, 
Can be done at 
etuiferenees, ser\iec 
site, etc. 



Works if ['larlicipani: 
and instructor arc at 
same get^graphic 
location (e.g., 
university campus), 
('an cover broader 
issues of digital 
reference (see Ask A 
I'rainiiig Mi'del 5 ). 
Praenee is authentic 
only if trainees \u\c 
access to nceessar) 
leclinoloiiv. 



Allows attention to 
indi\ idual pariieipants' 
needs and skills. 

Works in face-to-face 
or distance situations 
along u ilh print 
materials, clceironic 
materials, or both. 



lincapsiilatecl programs 
for indi\ idual 
instruction; praenee is 
not authentic unless 
integrated with Web or 
e-mail comp(uienl. 
Developmcm of 
product IS liir.e and cost 
inicnsu e. 



Job aids are siipplemeiUal resources intended to help the trainee erime up in speed during training ami to 
provide support for performanec (m-ihc-iob 
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As previously mcniionod, training programs can include a combination ordilYcreni 
materials, activities, and delivery tools. Some possible combinations based on training 
programs of existing AskA services include ihe following (see “Sample AskA Training 
Models" for examples of each scenario}: 

1. Text-Based Only (Responses Screened After Training). Manuals or other materials 
are available online or distributed \ ia e-mail to training participants. Performance of 
participants (e.g., question answering, question routing) is monitored after training, 
on the job. {Sample Ash\ Trainint^ Model L) 

2. Text — Practice — Feedback. Manuals and other materials (e.g., job aids) arc 
available to participants. Following an opportunity to read the materials, participants 
arc directed to or receive practice activities (e.g., practice questions to whieh trainees 
are expected to respond according to training materials). Service administrator offers 
feedback on practice activity performance. Practice-feedback cycle is repeated until 
trainee performs task successfully and is ready to participate in the service. {Sample 
AskA 7'raini}}i^ Model 2.) 

?\ Text — Shadow and Observation — Practice — Feedback. Training participants are 
given general training materials and then arc able to observe current staff performing 
job tasks. Training participant then responds to practice activities (i.c.. practice 
questions) and receives feedback from current staff members until ready to 
participate. {Sample AskA 'rrainin^ Model S . ) 

4, In-Person Clas.s — Text — Practice — Feedback. A group of parlidpanls attend an 
organized class led cither facc-to-facc or v'irlually (e.g., chat, video conferencing, 
Web, e-mail). Participants read training materials and then complete a series of 
practice activities and rccci\c feedback from current staff members. {Saiaple Asb\ 
Tniiubp^ Model 4 . ) 

Another mode of communication during training includes listscrv' discussions bciuccn 
participants and trainers. These discussions can be used to clarify areas of confusion and 
share specific information such as reminders, schedules, etc. Lislscrv discussions arc also 
a good way to document imporianl information about the training program itself, such as 
reactions of participants to certain activities and materials, 



Create Learner Assessment Plan 



During the planning phase it is important to decide how trainees will be assessed; in other 
words, how will you know when a training participant is ready to perform successfully as 
an infoimation specialist or AskA staff member? This issue may he most prominent in 
training sessions for information specialists who arc c.xpccted to perform the defined task 
of responding to user queries. Some services do not allow trainees to pass the training 
program until they demonstrate proficiency (e.g., they respond to practice questions 
successfully); others allow their information specialists to begin responding to queries 
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right away, but the responses are monitored by other staff members. (Some service 
policies require that all outgoing responses be monitored, while some allow responses to 
bypass the screening process once the information specialist is deemed ready.) 

Whatever type of assessment plan is selected, it is imponant to communicate to training 
pailicipants the criteria and methods by which they will be judged. When assessing 
information specialists’ responses to user queries, a number of different criteria may be 
used including the following: 

• Inclusion of all response components 

• Adherence to established turnaround time 

• Use of appropriate lone and language 

• Accuracy of facts and resources included 

• Relevance to question 

• Visual appeal 

• Adherence to specific service policies. 

In addition to judging trainees’ performance for the sake of determining readiness for 
participation, it is important to communicate to trainees how well they are achieving 
goals and what they can do to improve. Some services use a structured grading scale to 
inform information specialists as to the quality of their responses; others provide open- 
ended feedback to trainees with constructive criticism and suggestions for improvement. 
See “Sample AskA Training Models” for examples of practice and feedback options. 
More detailed discussion on providing feedback is found in Module 3.4: Implementing 
and Managing. 



•X. 
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Module 3.2 Summary 

This sccnoii covers the steps and deci.sitms necessary tor crcalinu a tram mg plan. 

• 'ITe planning phase is broken down into tour main steps; establish training goals, select and organi/.e 

training components, select delivery metiuuls, and create a learner assessment plan. 

• louring the “establish training goals” step, you w ill do the tollow , g: 

1 . Identify tasks and skills that are required for information specialists and stalf to carry out 
responsibilities. 

2. Analyze trainees to determine how well intonnaiion specialists and support siatT arc prepared to 
perform required tasks, 

3. Set training goals that outline what the training program will attempt to acetnnplish in terms of 
preparing information specialists and support stalf to perform required tasks. 

• During the “select and organize training components” step, you will do the following; 

1. Identify tihjeeiives tiiat the training audience (information specialists, su[>port staff, or both) need 
to perform in order to achieve the ultimate gt)ul. 

2. Select a set i>f ’‘treatments” (content, materials, and aetiviiics) that outline liow informatitni and 
ideas should be eommuiiieated to the training audience. 

• During the “select delivery methods” step, you will do the following; 

1 . Select the combination of products and events that make up the overall training program. 

2. Select tools to deliver training (c.g., communications technology, instructors, publishing 
environments, etc.) 

• During the “create learner assessment plan” step, you will do the fi^i lowing: 

1 . Identify eritoria to measure how well learners achieved ohjeeiises. 

2. Identify methods to test learners and provide feedback. 
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3.3 Producing the Training Program 

Once a iraining plan is esiablishcd il is lime lo pm the plan into action. Ai this point, you 
will ha\’C set goals for participants of the iraining program; selected objectives and the 
content, materials, and activities lo help participants reach objectives; determined the 
training products and events and the tools to deliver training; and established a plan Ibr 
evaluating training participants during and following training activities. 

T'his section describes the second phase of the overall process for creating a training 
program for AskA service staff: producing the materials and activities for a training 
program, preparing for delivery, and testing the program to determine its effectiveness. 



Create Training Materials 



Depending on the training plan, specific materials ma\’ be created or compiled from an 
existing set of resources. For example, a service may create a handbook lo guide the 
trainee through training activities and will refer lo existing materials for support: 
question-answer policy and procedure manual, lists of helpful Web sites, etc. 

There arc some impoilanl issues to consider when creating any type of materials that arc 
intended to be instructional in nature (i.c.. intended lo assist someone lo learn new skills 
or knowledge). This section outlines the following issues and pros'idcs examples. 

Scf Expectations 

Stale objectives of the training program dearly and early on in the program. (As 
previously discussed, il works best lo use the objccli\'cs lo guide the content of the 
overall program, so these ideas will be reinforced throughout.) Ohjeciivcs can be 
presented in training materials in a variety of ways. The following example presents 
sample objccti\‘es for information specialists: 

When you fir sh this training session, you will be able to respond to an incoming user 
question by 

1. incorporating all required response components. 

2. Using service resources and external Internet resources to research answers to user 
queries. 

3. Using your own e-mail program to receive, compose, and send messages. 
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The following example presems sample objccii\cs I'or support staff: 

Following this training session, you will be able to 

1. Use centra! system to review and route incoming questions to appropriate information 
specialists. 

2 . Review information specialists' responses according to set of criteria outlined in service 
guidelines for responding to inquiries. 

3. Provide feedback to information specialists using appropriate language and tone. 

4 . Locate Internet and other sources to include as references (supplemental information) in 
previously written responses. 

Crc'iUt' Positive Attitude 

U i.s important that the lone of the overall training program be positive and motivating. 
Language should be friendly and light, and information should be clear and concise so 
that the overall training experience i.s as pleasant and lime efficient as possible. Since 
many information specialists and support slLiff work as volunteers, it is also imporlanl to 
let staff know how much their eflbris arc appreciated. 

Moliv'alional components in training materials and communications can include these: 

• Altcniion-gctling language (humor, anccdolcs, information phrased as qucslicMis) 

• Visuals (graphics, illustrations, charts, bullet lists) 

• Inlcracii viiy (space for notes in materials; opportunities for discussion on c-mail lists) 

• Confidence-building language (c.g.. "Don’t worry if you have trouble searching the 
Web at first. It lakes practice!") 

• Acknowledgment of participants’ busy schedules and professional expertise 

• Description of rewards that staff can gain \vo\n participation in serv ice: 

0 intrinsic (c.g., personal satisfaction, professional experier ) 

0 extrinsic (c.g,, information rc.sourcc products, promoliona items, salary, etc.) 

ILxamplcs for creating a positive atliludc during training include the following: 

Thank you for participating in Ask An Engineer! As professional engineers, you bring your 
talents and expertise to this special program. Your volunteered time and services are 
appreciated! 

Following this training session you will join the other members of the Ask A Librarian 
team in fulfilling personal and professional needs and gaining important reference and 
technology skills! 
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Provide Clear Infonnaiiou and Examples 

Stale concepts clcariy and use contexts liiat arc familiar to the intended audience. Offer 
authentic, specific examples to illustrate concepts such as including required response 
components and providing feedback on information specialists’ responses. Examples will 
not only help to illustrate important ideas but will help familiarize trainees with the types 
of situations they will face as information specialists or support staff. Examples for 
providing clear information and examples in training materials are below. 



The following example illustrates a section from a manual for information specialists of 
an Ask A service for K-12 siudcnis: 



In responding to student questions, stress the importance of using a variety of information 
sources. It is important to remind students that the Internet is not the only place to look 
for answers to their questions, and that they may have to consult a few different 
resources. For instance, an encyclopedia is sometimes the best and fastest source. 

You might say something like this in your response: 

. . . Although there are many Web sites about hurricanes, a simple article in a genera! or science 
encyclopedia will best answer your question; How are hurricanes formed?’ Ask your school library 
media specialist to guide you to an encyclopedia or to other resources on hurricanes in your school 
library media center . . . 



The next example illustrates a .section from a manual for AskA service moderators that 
explains the process for screening incoming questions for distribution: 

In order for a question to be distributed to an information specialist for a response It must 
be 



• Comprehensible 

• Related to the subject of science 

• Unable to be answered by reviewing service resources 

• Unable to be answered by reviewing known Internet resources. 

For example, the question “What should I do for my science fair project?” should not be 
sent to an information specialist, because the student can find an answer by reviewing 
some of the sites recommended on the service's “Science Fair Link" page.^^ 









( 

\ 



This example was adapted from the MADSci Moderator's Mamat: 
htipi/Mw'w. madsci. ors/cisk_cxpert/moderators. html 
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Provide Oppornoiilies for Practice and Feedback 



'I'his is executed din'ercnlly by dilTcrent services. Many services extend pn.cticc and 
feedback past the training’ program by simply assessing staff performance on the job (e.g., 
information specialists” responses ♦ * user questions, etc.); in fact, some services conduct 
practice and feedback only after training (sec “Sample AskA Training Models"). 

It is advantageous to incorporate a practice and feedback component in AskA training 
programs for scx'cra! reasons including these: 

• Participant performance indicates whether or not they arc ready to perform their new 
roles and tasks successfully. 

• Feedback from service administrators or trainers informs trainees of ways to improve 
their work (e.g., responses to service users). 

• Results from practice activities serve to judge the effectiveness of the training 
program (e.g., if all participants .struggle with a certain task during a practice activity, 
it may be necessary to present the paHicular information in a different way). 

See Module 3.4 for tips on providing feedback to training participants. 

'I'hc following example includes a description of practice activities iVom an online 
training manual for information specialists of an AskA service for K-12 students: 

Once you’ve reviewed this training manual, you are ready to practice responding to some 
sample student questions. When you are ready to receive your first practice question, 
please complete and submit the form below. Your second practice question will be sent 
to you after you receive feedback on your first response, This process will continue until 
you and your trainer feel that you are ready to join an existing volunteer team. 

The next example includes a training agenda for in-house staff of information spcciali.sls 
(System-wide guide to AskFRIC question answering, 1998). 

• Review training manual. 

• “Shadow” veteran information specialists to observe searching methods and 
response preparation. 

• Complete exercises to practice using e-mail program and Internet browser. 

• Begin trial period where responses to user inquiries will be monitored and feedback 
will be offered on content, format, and tone. 

Sec “Sample AskA Training Models" for more examples of practice and feedback 
acli\'ities for AskA training programs. 
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i^rnvidc Posi-'irainiuy, Help and Reinforcement 

Some imining [Kirlicipanis may fiiul ii LlilTicull lo remember all ol ihe imi'KH’luiU 
informalion presented dui ina lrainin^> ft is a good idea U) provide "job aids’*- - handy 
reminders of how to perform tasks and supplemental i'est)urees that parliei|)ants can refer 
to after the training prc'igram. Hxantples of job aids may include the 1‘ollowing: 

• ('lieeklists 

• Diagrams 

• Lists (d’ additional resources (Web links, bibliographies) 

• Directions for using required tools (e.g.. databases. c|iiestion-routing s(d'tware) 

• Lxemplary work (or non-examples). 

In addition to resource-based job aids, sers iees may offcraeeess to staff members who 
can serve as mentors to new' parlieipanis folknving the training sexssion, This type ol‘ 
airangement allows newv staff members lo case into tlicir ixdcs by relying on support from 
ttiose who arc more familiar with the process. 

'Hie following example includes a checklist of response components as one Job aid: 

Hare is the list of components to include when writing responses to users: 

1 . Include original message. 

2. Greet user by name. 

3. Short factual response (if appropriate), 

4. Reference to resources used to answer question, 

5. Directions for finding resources. 

6. Name of information specialist. 

'rhe following example lists suppicmeiilal resources for learning lo use search engines: 

Below are some good resources to help you choose the best Internet search tools for 
particular situations: 

Sink or Swim: Internet Search Tools & Techniques by Ross Tyner, M.L.S., 
Okanagan University College 

hitp://oksw01. Okanagan. baca/libr/connejt96/search.htm 

Understanding WWW Search tools htip://www.indiana.edu/--librcsd/se3rch/ 

Web Searching Guide from the Internet Public Library 
http://lpf. sils. umich. edu/refAA^ebsea rching. html 

SCOTT http://www.askscott,com/ 
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Prepare Delivery Tools 

Depending on ihc combination ofdclix'cry tools selected, prcpu.ation will consist oT 
dilTcrcnt types of actions and decisions. By this stage, delivery tools will have already 
been selected based on several issues including accessibility and skills of participants (see 
‘‘Select Delivery Methods'' in Module 3.2). At this stage, delivery tools arc prepared for 
production and implementation of instructional materials and activities. For each type ol' 
delivery tool selected, it is important to consider the following issues: 

Tcduiical Issues for Ask/\ Service and Panicipanis 

These can include development issues such as establishing Web space for an online 
training manual or implementation issues such as making sure a classroom is wired ‘or 
Internet connection or that all participants’ names arc added to the electronic di.scussion 
list. 

Skills Necessary for Individuals Developifii^, Implenicniifiy, and Manai^ini^ Idols 

it is a good idea to make sure that everyone involved in preparing and using selected 
delivery tools has the necessary c.xpcrtisc or can be trained in the required areas. This can 
include basic writing skills, proficiency with Web site design (e.g,, HTML, Web editing 
software, graphics design, etc.), or c.xpcricncc in listscrv management. 

Aesthetic Possibilities and Liffiitations ofl'ooHs) 

Some tools allow more flexibility in design than others. I’raining materials intended fore- 
mail distribution on UNIX-based systems should include only those formatting options 
available for text-only (or ASCII) documents. Therefore, more creativity is needed to 
make the document look visually appealing (e.g., adequate white space, borders with 
asterisk marks or dashes in between sections, etc.). Materials distributed on Web pages, 
however, may include brightly colored graphics, animation, and multiple font types. 

See Table 3-5 for c.xamplcs of issues to consider when preparing different types of 
delivery tools. This table docs not represent a comprehensive list of issues. 
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Table 3-5 Issues in Preparing Delivery Tools 



Delivclrv ^bols 




Technical Issues Skills 



Aesthetic Issues 



Interactive 
communications 
environment (MOO, 
IRC) 



Print-based materials 



I list rue tor- led 
presentation or class 



Onc-on-onc mentoring 



Secure Web space 



Collect participants' 
e-mail addresses 



Do administrators and 
participants have file 
transfer software? 
Which programs and 
versions are participants 
using? 



If text-based, users need 
access to UNIX server 
and directions for 
entering chat 
environment. If 
graphics-based (using 
specific software 
package such as 
The Palace 

{hftp://w\\'\v.pa!acespacv 
.com/ ), service staff 
must create training 
environment and users 
must have necessary 
software to participate as 
well as passwords and 
directions for entering 
environment. 



Copies; printing and 
binding; mailing and 
distributing 



Check on availability of 
equipment (overhead or 
LCD projector), Internet 
connection, etc.; chairs, 
lighting, etc. 



If face -to- face, do both 
people require access to 
computer? Physical 
space for both people; 
establish participant- 
mentor partnerships 



Knowledge of 
HTML or Web 
editing software 



List management 
(knowledge of 
listserv or other 
discussion list 
software) 



Knowledge of file 
transfer software 



Manage or guide 
chat discussions 

Create environment 
for graphical chat. 



Writing and 
formatting 



Presentation or 
leaching skills; use 
of presentation 
softw'are; operation 
of equipment 



Interpersonal skills; 
ability to 
accommodate 
training delivery to 
individual needs. 



Balance of text, graphics, 
animation (depends on 
identified technical 
capabilities of participants) 



Must use creativity to present 
information for those 
participants using UNIX 
systems (white space, 
borders). 



Create documents according to 
capabilities of specific 
programs. 



Text- based communications 
environments are limited in 
terms of design, 



Software for graphical 
communication allows wide 
selection of graphics, 
animation, sound, etc. in 
creating special environments. 



Balance of text and graphics 



Presentation software offers a 
variety of opportunities for 
graphic and text design. 
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Pilot Test the Tr nlng Program 

In order lo determine whether or not the training; materials and activities will be 
successful, it is helpful to conduct a first run of the training program on a pilot group. 
This works best when members of the pilot group intend to serve as staff for the service. 
The only difference betvvcen a pilot test and an actual training session is that the pilot lest 
is used to pay special attention to the effectiveness and efficiency of training materials 
and events and learner performance. The point of the pilot tests is to catch any major 
flaws before final drafts of materials are produced or major decisions are made about 
tools and events. Results from pilot tests can save services money, time, and other 
resources. See Module 4 for issues in pilot testing the overall AskA service. 

Questions asked during a training program pilot lest may address the following topics: 

• Effectiveness of materials 

0 clarity c.ad comprehensibility of content 
0 visual appeal of materials 
0 helpfulness of practice activities 

• Efficiency of overall session 

0 amount of time spent on training 

0 ability to use the technology necessary to participate in training (e-mail 
program, Web-based database, etc.) 

• Extent to which trainees are prepared to perform staff duties 

Questions are most effective when they are designed to elicit more than a “yes” or “no” 
response. For instance, you can include the following: 

• Multiple-choice responses (“On a scale from 1 to 4 where i = ‘not at all prepared’ 
and 4 = ‘very prepared,' please indicate how you feel about responding to user 
questions at this time.”) 

• Opportunities for open-ended responses (“What additional information do you think 
should be included in the training materials?”) 

During and after a pilot test, different evaluation methods can be used lo determine 
results. Some services may distribute instruments such as questionnaires via e-mail or the 
Web (see Figure 3-4), while others may hold informal conversations on an e-mail 
discussion list. 
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P'igurc 3-4: Training Pilot Test Questionnaire Distributed Via K-mail 

Informalion Specialist Training Session 1 
Questionnaire 

Now that you've completed the training session, we hope that you will share your 
comments as to how the training materials and activities can be improved. To answer this 
questionnaire, just choose your “reply” command and the “include original message" 
option and write your answers below each question. Names will be stripped from your • 
messages before wc read your responses, so feci free to be totally honest. 

Iliank you in advance for all your help! 

1. Please rate the following aspects of the training materials and activities based on the 
scale below: 

Scale: 

(4) Very helpful 
(3) Helpful 
(2) Somewhat helpful 
(I ; Not at all helpful 

• Service policies 

• Tips for responding to user questions 

• Links to Internet sites on Web searching 

• Checklist of response components 

• Exemplary response 

• Practice questions and feedback 

2. Please comment on any aspect of the training materials or activities listed above. 

3. Overall, was the information presented in the Online Training Course clear and 
understandable? Please explain. 




4. How prepared arc you to respond to user inquiries by following the steps presented in 
the training materials and feedback to your practice questions? 



5. What additional information do you thirk should be included in the trai..ing 
materials? 
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Following the pilot test tmcl data collection, it is important to revise any aspects ol the 
training program based on results. Although the pilot lest is dedicated to identifying 
problems in the training program, each training session should scr\c as an opportunity to 
identify nev\' problems or area.s to be re\'iscd. Sec Module 3.4 for more information on 
ongoing training e\'aluation techniques, 

Once the pilot training program has been revised, you are ready to roll! The next section 
discusses methods for implementation as well as ongoing management of an AskA 
training program. 
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.Module 3.3 Summary 

Tills seclion cox ers the steps inxohod in putting the training plan into aeiion. 

• Tlie production phiise consisis of three main steps; create training malerials, prepare delivery rools. and 
pilot test the program. 

• \\'hen ercaling training materials, it is important to consider the following issues: 

1 . Inform trainees of what ihcx' should accximplish by the end of the training program in terms of 
training objectives. 

2. Create a positixe altitude by using attention-getting and eonfidencc-building language and 
attractive visuals; prtividing opporiunities for interaction; acknowledging participants' busy 
schedules and c.\porti.>c, and describing rewards that can bo gained from participation, 

3. Provide clear information and examples within the ‘ v\t in ordet to stress important concepts 
and to familiarize participants with the ultimate work eontc.xt, 

4. Prox ide opportunities for participants to practice job ta.sks, and offer consirueli ve feedback on 
parlicipani performance. 

5. Provide resources and support to guide participants m required tasks after the training 
program. 

• When preparing delixery ltH)ls, il is important to consider the l\>llowing issues: 

1. Technical i.ssiies of .Ask,-\ serx ice and participants (c.g., secure Web space, make sure 
lechnology is compatible xviih participants* sx-siems) 

2. I*'\perlise neecssarx for individuals developing, implementing, and managing tools (e.g,, 
knoxx ledge of HTML, lislscrv management) 

3. Ac.slhclic possibilities and limitalions of the lool(s) (e.g., information transmitted x ia e-mail 
should include a large amount of xxhue space and UNIX-omipaiiblc symbols) 

• When preparing for and implementing a pilot test, it is important to consider the folloxxing issues: 

1 . Questions addressing effectiveness of materials, efficiency of the overall session, and extent 
to xchich trainees arc prepared to perform stuff duties. 

2. Methods for eliciting responses to ilie training program, such as questionnaires or clectmmc 
discussion. 

3. Items to be revised before producing the final draft of materials or making final decisions 
about training activities. 



113 j-, 

I I 



best copy available 



Training 



Module 3 



3.4 Implementing and Managing the Training Program 

At this point, you should be ready to begin the training program, liainmg mateiials have 
been developed and revised, and delivery tools have been prepared for use. This section 
discusses important issues to consider in implementation and ongoing management of the 
training program. Issues include scheduling training events, providing feedback to 
trainees and staff, and evaluating and revising the training program. 

Schedule Training Events 

This is the step of the process that most directly affects the participants: training materials 
arc distributed, electronic discussions are held, practice activities are conducted, etc. 
Although most of the preparation is handled during the production stage, it may be 
necessary at this point to organize the actual schedule of events. Below is a typical AskA 
training program schedule with examples from different AskA services (see ‘'Sample 
AskA Training Models” for more detailed descriptions): 

Rccmitiny, and Registration 

Most training sessions occur just after an individual or group of new participants 
becomes part of the service staff or expresses interest in participating. Methods for 
joining the service as an information specialist or staff member include 

• Filling out a Web signup form 

• E-mailing the service administrator 

• Joining the service as a staff member (through interviewing process, etc.) 

• Registering for a university course. 

A screening process may also be involved in order to maintain quality of the service and 
to meet service guidelines. More information on recruiting new participants for an AskA 
service is found in Module 5: Contributing, 

Besides recruiting and signing up new staff members, you may need to make special 
arrangements for the training program such as setting up e-mail lists for groups of 
participants and confirming training start dates. 
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Information Presentation and Orientation 

Services may inilialc Lhcir training sessions by presenting the schedule of activities 
followed by distribution of a written manual (cither online or in print) that explains 
service policies and outlines job tasks (see Table 3-2 for content frequently covered in 
AskA training materials). Those services that rely on a written manual as the primary 
training tool simply point new staff members to the material to begin the training process. 

Some services also present information through discussion — either electronic or face-to- 
face, group or individual. At this point, trainers (staff members delivering the training) 
can offer advice, and participants can ask questions about specific aspects of their future 
job tasks. 

Practice and Feedback 

Practice and feedback activities normally take place after participants are familiar with 
the information presented in materials and discussion. As previously discussed, this 
aspect of training is implemented differently by different services to the point where 
some services do not include practice and feedback as a part of formal training but 
closely monitor staff work on the job. For instance, some AskA services monitor all 
outgoing responses from information specialists before they reach the user, while others 
periodically spot-check responses from information specialists who have successfully 
completed practice responses during a previous training session. 

The elimination of practice and feedback during training quickens the process and allows 
information specialists to start answering questions immediately; however, it adds time to 
dady management tasks and decreases the chances for individual improvement unless 
detailed feedback is offered for each response. (See discussion on providing feedback to 
trainees and staff later in this module.) 

Graduation and Transition 

For some services there is a defined transition between the training program and actual 
service participation (i.e., trainees successfully complete training activities and then take 
on their roles as service volunteers, experts, moderators, etc.); for other services, the lines 
between training and service participation are less defined. 
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Below arc some examples of iransilions involved in training programs for AskA service 
information specialists: 



• Training participants complete a series of practice exercises: they reeci\'e fictional 
user questions and arc expected to respond to them as they would actual user 
questions. After successfully completing the practice responses, they begin 
responding directly to incoming user questions. Responses arc monitored periodically 
after being sent to the user. If an information specialist proN’idcs quality responses 
over a period of time, he or she may be asked to monitor others' responses after they 
are sent to the user. (See “Sample AskA Training Model 3".) 

• Training paiTicipants respond to incoming user questions in the role of actual 
“experts.'’ During training, responses arc monitored by “trainers" (e.g., current 
service staff members) before being forwarded to the user. This process continues 
until the trainers determine that the trainee is ready to bypass the monitoring stage 
and respond to users directly. (See “Sample AskA Training Models I and 4.") 

• New information specialists immediately begin responding to incoming user 
questions in the role of actual “experts" without a formal training program. All 
oiit^oiny^ responses are nioniforecl by service fnoderators. “Experts" wlio provide 
quality responses over a period of time may be asked to monitor other expeiis’ 
outgoing responses or may be given other responsibilities or perks. (Sec “Sample 
AskA Training Model 2.") 

As illustrated above, there arc a variety of options available for “graduating" trainees or 
new staff members into their ultimate service roles. There is no one perfect way to 
implement this aspect of an AskA training session, but the right balance of events and 
activities can increase efficiency of service operations. A paiiicular AskA service should 
choose the order and nature of transitional c\'cnts according to the following factors: 



• Need for new AskA staff. Is it desirable to move people through the training process 
quiekly so they can cairy out their roles as soon as possible? If so, the training 
process might allow information specialist trainees to “praetice" responding to real 
ineoming user questions. These responses mus! be monitored by knowledgeable staff 
before being forw arded to the user, whieh could end up taking more time in the long 
run. 

• Availability of current staff to implement training. Arc there enough people 
a\ailablc to o\’crscc training activities and pro\ idc feedback in trainees (one 
experienced staff member to oversee two or three trainees or new staff members)? Is 
there an individual or team of staff members dedicated to training? If there is enough 
time and attention devoted to training, it would he possible to conduct the training 
process meire quiekly. 
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• Importance of quick response turnaround. If service policy promises quick 
lumaround on responses, there may not be enough time for trainees or new staff to 
‘'experiment” with real user questions. The time taken for staff to monitor responses 
before they go to the user may exceed the required time limit. In this case, it is better 
for novice information specialists to practice on fictional questions until they have 
achieved training goals and objectives. 

• Difficulty and extent of skills to be learned during training. If information 
specialists and staff are expected to carry out an extensive set of tasks or must learn a 
set of new skills or knowledge (e.g., use of technology, subject knowledge, research 
skills), it w'ill be advantageous to devote a significant amount of time for formal 
training. This will result in fewer mistakes and less monitoring in the long run. 



Providing Feedback to Trainees and Staff 



Providing feedback is one of the most important aspects of a training program. It offers 
each participant personal attention and constructive support for improvement. Some 
feedback may be oral (for instance, in a facc-to-face context) or written; written feedback 
may include open-ended comments or symbols from a grading scheme (e.g.. 5 - more 
than expected, 1 = extensive revision necessary, etc.). (See “Sample AskA Training 
Models” for examples of specific feedback methods.) Regardless of the method and 
timing of delivery, feedback should both guide and motivate participants and staff to 
perform tasks successfully (as defined in training objectives). In order for feedback to be 
helpful, it should include the following components: 

Positive Statciueni 

Start the conversation by praising the trainee or staff member for some positive aspect of 
his or her w'ork. For instance, highlight the information specialist's friendly tone and then 
point out the fact that he didn’t include the user’s original message in the response. 

This was a good try for your first response. The Web site you 
suggested does a great job of explaining the history of the typewriter 
keyboard. The illustrations are also very helpful. 

You expressed genuine interest in the student’s question. This can 
help to motivate students. It was also a good idea to suggest Infotrac 
and a local physician in addition to Internet resources. 
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Objectives 



Remind trainees or staff of the objectives they should achieve, including components to 
include in user responses and other specific service procedures, noting which ones were 
achieved and which ones were not. 



You included almost all of the components of the response checklist: 
you included the original message, greeted the student by name, 
included a factual response, and cited your resources. One thing you 
left out was the archive account address in the carbon copy field. 

It also helps to explain the importance of including certain components to link training 
objectives to real AskA service situations. 



It is important to include the search engine and search terms you 
used to find the Web sites so users can learn to conduct Internet 
searches on their own. 

Tips and Siii^gesiions for Improvement 

Sometimes trainees or staff members have difficulty performing certain tasks because 
they are still new to the required technology or other aspects of the Job. Although related 
information may be covered in the training materials, it is still helpful to reinforce or 
expand on specific concepts in response to mistakes or questions of individual 
paiticipants. 



Include original message. This can usually be done by selecting 
options for “reply” and then “include original message.” Check your 
e-mail program for directions. 



Instead of typing Web addresses in your responses, another 
alternative is to copy the URL from the Web site and paste it directly 
into the message — if you have the capability to keep your e-mail and 
Web browser open at the same time. This will help to avoid incorrect 
Web addresses due to typographical errors. 
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In addition to providing tips for performing certain technical tasks, it is helpful to provide 
general suggestions for improving one’s work, such as responses to users. These 
suggestions may address the aesthetic or educational value of service products. 



One suggestion for making responses easier to read is to physically 
separate ideas from each other. For instance, start a new line when 
talking about a new resource, like this: 

• Bill Nye Episode Guide: DESERTS 

http://nyefabs.kcts, org/nyeverse/episode/e 72, html 

• Deserts 

http://www.cuug,ab.ca:8001/-^animal/deserts.htmi 

For this particular question, ! would suggest referring the student to 
the archive. Some of our math experts have already written excellent 
responses on the order of operations that could serve as 
supplemental resources for the student. 

Availability of Support 

No mailer how well the irainee or new staff member is performing, it is imporlani to let 
each participant know that an administrator or other staff member is available for help on 
an ongoing basis. This communicates to staff that the service is always looking to 
improve and that individual ailention and support arc available when needed. This can be 
done by including a quick statement al the end of a feedback message (c.g., ’‘Let me 
know if you have any questions.”) 

Ongoing Evaluation and Revision of Training Program 

As a service matures, training materials and programs should be altered according to 
feedback from information specialist^ and support staff (e.g., common questions or 
confusion about particular processes), ■ bservation of staff performance (e.g., quality of 
information specialists’ responses), and other factors relating to user satisfaction. 
Developing a training program is an evolving process that responds to needs as they arc 
identified. Evaluation and revision of the training program will complement the overall 
evaluation efforts of the AskA service (see Module 6). 

The ongoing evaluation of the training program is an extension of the pilot testing 
process (see Module 3.3). The difference is that an ongoing evaluation process can 
address the impact of training materials and activities on staff* performance over a period 
of time. This provides an opportunity to monitor work such as responses to user questions 
and identify common issues among different staff members and situations. The following 
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issues (as well as others) may signal the need for evaluation and revision ol' training 
materials and activities: 

• Staff performance (e.g., lack of a particular component in information specialists’ 
responses, inability of moderator staff to provide effective feedback to information 
specialists) 

• Development of new tasks (changes in procedures) due to new software, improved 
processes, etc, 

• Feedback from staff (c,g., after some experience, staff members may discover they 
v\'ould be more comfortable or more successful performing tasks if certain skills were 
developed) 

Evaluation Methods and Tools 

Fvaluation methods for AskA training programs can range from formal plans (e.g,, a 
focused assessment of training materials, activities, etc.) to informal activities (e.g., day- 
to-day observation of staff, unsolicited feedback from users). Figure 3-5 is an example of 
a Web-based questionnaire used to elicit feedback from information specialists following 
their participation in a training session. 

Figure 3-5 ^ruining Questionniiire on tlie Web (sample excerpt } 

Information Spcciali.st 'rraining Questionnaire 

It is lime to revise our training program. Now that you've worked as an int'ormatnui specialist for a lew 
nicmihs, wo hojie that you will sliarc ynur comments as to how training materials and activities can be 
improved. To answer this questionnaire, just complete the questions below and click tlie “submir 
button at the bottom of the page. 

Thank you in advance for all your help! 

1. How well did the training materials and activities prepare you to do the following (please rate on a 
scale from 1 to 4 whoie 1 = very well and 4 = not at all): 

respond i(i user questions: 

u.sc service information rc.sources: 

use service work- How software program: 

2. What additional tasks or issues do you feel you were unprepared for upon beginning uork with the 
service? 

3. What information would be most u.scful to you now in completing your required tasks? 
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Once information is oollcclccl frt)m the iniining program evaluation, the data should be 
anaiy/cd. Services can re\'ise programs as necessary. Depending on evaluation results, 
services may revise training manuals to include further explanation ofecrlain procedures 
add more opportunity for practice during the training program, or adjust other iraming 
components to better accommodate staff in effectively preparing for (asks invQ)lved in 
AskA service participation. Sec Module 6: Evaluation, for more suggestions on 
eonduciinu evaluation of the overall AskA service. 



Module 3.4 Summary 

This sociion co\ ers the slops imoivcd in im[ilonicniinu and managing llio naming program on an ongoing 
basis. 

• Ongoing impicmontannn and managcmcni of an Ask.'N (raining prtigram iinobos scheduling training 
e\onis, pro\'iding t’ocdhack (o trainees and slalT, and evaluating and re\-ising the iraming program. 

• Many sorMces include the t'oli(n\ ing training cxenis in the implementation of their training programs; 

1 . Recruiting new' stall' and “registering" them for participation in the service or training 
program 

2. ^^’esenting information (training content) and orientation to the scr\iee (e.g., policies and 
procedures, etc.) 

3. Practice activities and feedback on performance 

4. “Graduation." or a transition fnan training or trial status to official staff member or v(duntccr 
of the service, 

• liffcclixc feedback from cxperieiiced staff members guides and iiKMivates trainees and new staff to 
inipnue their pertormance. The folknvmg components should be included m feedback 
commuiiicaliins: 

1 . Statements praising positiv e aspects of the work 

2. Reminder of objectives outlined in training and siaicmcnt of progress m terms of achieving 
the objectives 

3. TTps and suggestions for improvement 

4. Statements offering individual support during and after training. 

• .All ongoing esalualion and revision proce.ss allows the AskA scr\ ice to determine the impact of the 
(raining program or) staff pcrftirmancc and ways to imj'irovc eoniponems. Ongoing evaluation and 
revision involves the fidlowjng: 

1 . Identifying opportunities for ewaluaiion and revision 

2. Gathering evaluati\ e information through obscr\ ation, instrument distribution, etc, 

3. Revising training program based on evaluation results. 
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Sample AskA Training Models 

The following pages include descriplions of training programs of existing AskA services. 
These training programs serve as models for AskA service training, representing some of 
the possible methods for distributing training materials, conducting training activities and 
other events involved in preparing information specialists to answer user questions. 
(Methods for training support staff arc mentioned when available.) 



AskA Training Model 1 

Ask Dr. Math, The Math Forum, Swarthmore College 
htt|)://roruni,s\varthniore.edu/dr.math/ 



Text Manual 



Trial Run 



Training for Volnnieer Experts ("Math Doctors") 

Recruiting and Reg is fration 

The Ask Dr. Math Web site contains a section that introduces the service and links to a 
signup form to become part of the volunteer math doctor team. Potential pailicipanis can 
access this information through links from the main service Web site. For each new 
volunteer who signs up to become a math doctor, an account is created in the service 
system. 



Information Presentation and Orientation 



l:ach new volunteer receives a message from service administrators that explains basic 
information about the service and asks the volunteer to review the online manual, “Guide 
to Writing Responses'’ (Foster, 1998). This guide covers technical issues (c,g,, format for 
including math symbols and graphics in responses, methods for .selecting questions from 
the queue for response, required components for responses); quality issues (c.g., writing 
clearly and with a friendly tone, explaining reasons for performing certain math formulas, 
fostering user creativity); and information on available resources (e.g., FAQ, archive, 
other sites), New participants arc encouraged to become familiar with service resources 
that can aid in response writing, including the archive and FAQs. 
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Practice and Feedback 

New “math doctors” begin answering incoming user questions but send their responses to 
an account called the “Holding Tank.” Responses in the “Holding Tank” are reviewed by 
experienced staff members or service administrators. Responses that meet service 
requirements are forwarded to the user; those which do not meet all requirements are 
returned to the “math doctor” for revision. If the math doctor is unable to revise the 
message, an administrator will rewrite it and add his or her name to the response. 

Trainers look for responses that contain clear writing styles and good answering 
techniques and reflect understanding of math concepts. They provide feedback to new 
math doctors by offering examples and explanations and encouraging conversation. 
Trainers keep notepads on each new volunteer to track their progress during the training 
process. 

Transitions and Rewards 



Volunteer notepads are reviewed periodically to determine which volunteers should 
receive “tenure” (the opportunity to send responses directly to the user). New volunteers 
who have consistently produced inadequate responses may be asked not to continue with 
the service. 

Traifiing for Adfuinis'trafors and Ollier Trainers 

Currently there is no formal program for training staff other than new volunteers. 
However, discussions are held about the training process, and the service may provide a 
train-the-trainer program as it expands its training staff. 



See Module 2 for complete Ask Dr. Math case study. 
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AskA Training Model 2 

MAI) Scientist Network, Washington University Medical School 
http://www.niadsci.org 

^ Responses Screened 
after Training 



Training for Volunteer Experts {Scientists} 
Recruiting and Registration 



Text Manual 
Only 



Individuals interested in serving as science experts complete a signup form on the 
service’s Web site. Information captured on the form includes contact information, 
professional or academic position, and areas of interest and expenise within science and 
math. Each form is reviewed by a service administrator who sends each candidate an 
introductory message containing information on the service’s Web site contents and 
question-answer procedures. 

Information Presentation and Orientation 



The official “training" process involves an online manual containing procedures for 
answering questions, administrative information (scheduling, etc.), and using World 
Wide Web resources in responses (links, searching for sites, etc.: MAD Scientist 
Network, 1997b). Trainees are also introduced to the online interfaces for submitting 
answers; this information contains additional material to suppoiT the response process 
following training in the form of a job aid. Important points stressed during training 
include expected response components and turnaround time (e.g.. respond w'ithin s /cn 
days after receiving a question or notify administrator of inability to field the quest. on). 

Ongoing Practice and Fe e dbac k 

Although the training process docs not involve practice activities, all responses undergo a 
screening process before they arc sent to the user. All responses from all experts arc 
reviewed by “moderators" for quality and appropriateness and then graded on a .scale 
from 1 to 5 (where I “great answer" and 5 = “requires heavy editing to be useful"). 
Grades are used for tracking an expert’s performance overall and within specific 
categories and for indicating quality of responses in the archive for weeding puiposcs. 
Grades are not shared with the individual experts. 
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Moderalors provide feedback to expeils when they notify them that their answers have 
been placed on the Web server. If the response is placed “on hold/' moderators inform 
expeils of the con'cciions or additional information needed for the response to meet 
appropriate standards. 

Transitions and Rewards 

Expeils who demonstrate quality resi)onses over a period of time may be asked to serve 
as moderators. They may also be highlighted on the service Web site on a page called the 
“MadSci Hall of Fame'’ (lutp://w\'\\\nKtdsciA\i{stic\ht/madliofJuml}, which includes 
exemplary experts' biographies, research interests, and other information. In the future, 
the service may be able to offer Web space to some expens so that they can develop their 
materials. 

Trainui^ for ^'Moderciiors " 

Moderators of the scrxicc review ail incoming questions to the service as well as all 
outgoing responses from the experts. Each moderator is responsible for the questions and 
responses in one or more subject areas. 

Moderator training consists of an online manual (MAD Scientist Network, 1997a) 
highlighting required background and expectations of moderators, time commitments, 
information on using the moderating interface, and tips for reviewing questions, 
reviewing and “grading” responses, and reviewing and recycling unanswered questions. 
Moderators are also referred to internal ser\ icc Web resources for becoming familiar with 
service. 

See Module 2 for complete MAD Scientist Network ease study. 
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AskA Training Model 3 

KidsConnect, American Association of School Librarians 
http://www.ala.org/ICONN/AskKChtml 




Training for School Libraty Media Specialist Volunteers 

KidsConnect volunteers are library media specialists who answer general subject 
questions from K-12 students. All new volunteers undergo a formal training process 
before they are able to answer incoming student questions. 

Recruiting and Registration 

Volunteers are recruited through postings to professional school library listservs, 
presentations at school library conferences, and professional publications. Dates of 
upcoming training sessions are included in some of these communications. Interested 
individuals contact the service coordinator indicating possible dates for participation in a 
training session. 

The coordinator organizes interested individuals into training groups of 5-*10 people and 
confirms training start dates with each individual. At the stait of a training session, the 
coordinator distributes an introductory e-mail message to each member of the session (via 
a distribution list) outlining the schedule for the session (e.g., read materials, receive first 
practice question, respond to practice question, etc.) and location of online training 
materials (along with login and password for accessing documents). Training participants 
are encouraged to send an e-mail message to the list introducing themselves in terms of 
their current professional positions (including grade levels served) and interests in order 
to build a sense of community among members. 

Information Presentation and Orientation 



The first activity of each training session involves reading the online manual and 
supplementary materials. The online manual contains information on the service, 
volunteer responsibilities, course objectives, required skills and technical capabilities, 
descriptions and examples of guidelines for responding to student inquiries, and 
directions for practice activities. Supplementary information provides tips on using the 
Internet as a research tool and responding to student inquiries. The supplementary 
document is intended as a guide during and after the training session. 
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Practice and Feedback 



On the day scheduled, the service coordinator distributes the first practice question to 
each participant. (If participants complete the online manual and supplement earlier than 
this scheduled dale, they can request the first practice question via a Weh-based form.) 
Paiiicipants respond to the first practice question (a previously asked student question) 
using the procedures outlined in the training materials. Participants send their responses 
to the service coordinator or other trainer who in turn provides them with detailed 
feedback. After their first practice response has been assessed, participants receive their 
second practice question from the coordinator or trainer (this is done on an individual 
basis). The practice-feedback cycle continues until participants’ responses are acceptable 
and participants feel comfortable composing responses. Training participants normally 
receive about three or four practice questions before being asked to join a team of active 
volunteers. 

Once a volunteer is part of an active question answering team, he or she receives periodic 
feedback from his or her team leader, an individual who mentors and manages a group of 
volunteers. Periodic feedback is less involved than feedback received during training: the 
purpose is to verify that required response components are included and that questions 
are answered appropriately. 

Transition and Rewards 



After trainees have successfully responded to three or four practice questions, they are 
invited to join an existing team of volunteers who answer questions from students of the 
same grade level on the same volunteer schedule. 

Volunteers vvho provide quality responses over a period of time arc sometimes asked to 
serve as team leaders, volunteers who oversee responses from members of a given team. 

Team Leader Training 

Team leaders are volunteers who have already undergone the main training program and 
have successfully answered student questions for some lime. Training them for their 
duties as team leaders consists of pointing them to a short online manual detailing their 
responsibilities and a guide for providing feedback to volunteers. 

See Module 4 for complete KidsConnect case study. 
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Ask A Training Model 4 
AskElRIC, Educational Resources Information Center 
http://wvvvv.askeric.org 




A 



Tni'uuJi^for Network Information Specialists 

AskERIC Network Information Specialists (NIS) provide responses to questions 
regarding educational theory and practice. NIS are located in several different 
clearinghouses of the Educational Resources Information Center (ERIC) system. This 
training model represents training activities in the central AskERIC office at the ERIC 
Clearinghouse on Information cS: Technology based at Syracuse University. 

Recruiting and Reaistration 

NIS arc hired through the ERIC system as paid staff of an individual ERIC 
clearinghouse. Upon stait of the position, each NIS is taken through an individual 
training process depending on individual clearinghouse practice. 

Information Presentation and Orientation 



Initial training consists of reading print and online manuals outlining service history, 
mission, and supporting organizations; question-answer policies and procedures; 
technology basics (e-mail, system software and hardware); database searching (service- 
specific rcsourcesj; and Internet searching (System-wide guide to AskERIC question 
answering, 1998J. (Materials addressing general system issues arc shared across 
clearinghouses.) In addition to reading training materials, NIS are given a general 
orientation to the specific work environment. 

Shadowing 

After the new NIS has a chance to read through the training materials, he or she shadows 
a few veteran NIS and observes the question-answer process firsthand. During the 
shadowing period, new NIS learn tips and techniques for question answering, including 
issues of tone and appearance of e-mail responses. 
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Practice and Feedback 

After the shadowing period, new NIS respond to questions on their own. For about two 
weeks, responses are reviewed by veteran NIS before going out to the user. Ongoing 
feedback is offered to NIS on a periodic basis. 



See Module 3 for complete AskERJC case study 



Ask A Training Model 5 

Internet Public Library, University of Michigan School of Information 

http://vvvvvv.ipl.org 



In-Person 

Class 




Text Manual 



Practice 



Feedback 



/ raining Jov hue met Public Libnuy Staff 

The Inlcmei Public Library (IPL) is a project of the University of Michigan School of 
Information that provides Internet users with access to resources as well as digital 
reference service. The IPL serves all audiences with special attention to youth needs. 
Online training resources such as policy manuals aie available to staff who maintain the 
project and professional librarians who answer digital reference questions. IPL‘s main 
training efforts have been spent on educating those external to the project about digital 
librarianship and IPL policies in order to recruit new staff and information specialists and 
to produce new IPL resources. This training effort is delivered in the form of a graduate 
course through the University of Michigan School of Information. 

Registration 

Students register for the digital librarianshij) course through the university. Many students 
who lake the class continue with the internet Public Library (IPL) service as a reference 
volunteer or staff member. 

Information Presentation 



The course is dedicated to developing resources and providing question-answer support 
for the IPL as well as discussing the nature of librarianship in a networked environment 
Face-to-face class meetings provide a I'orum to discuss IPL serv ice procedures and 
technology; skills and issues invoh'cd in digital librarianship; Internet searching; and 
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logistical issues regarding siudenl projects. The class has been taught via distance with 
initial face-to-face interaction and then continued through listserv discussion and 
individual project work (sec Practice and Feedback below). 

Supporting materials include information on question answering policy (preparing a 
response, mandatory response components, etc.); reference response style guidelines 
(c.g., providing instruction to the user, writing with a friendly tone); using the IPL 
question-answer interface: and tips for finding and evaluating Internet resources. 

Practice and Feedback 



The emphasis of the course is on practical experience that often occurs outside of the 
classroom: collection development projects, question answering, visiting the IPL lab. The 
main practice component of the course is the students’ question-answering experience. 
During the course, each student is responsible for answering 8-10 questions per week 
according to IPL question answering policy. Experienced IPL volunteers or staff team up 
with groups of students to provide support and feedback on the question-answering 
process as well as on individual student responses. This monitoring stage ends once 
students become regular volunteers (if they choose to do so). 
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F2lectronic Reference: Service Issues and Necessary Skills 

^ by Sara Weissman 

The Monis County (NJ) Library began as an experiment in electronic reference in 
January 1997. Intended as a six-month project, the service was so manageable that it has 
continued. In 1 4 months, volume has risen from one or two questions a week to two or 
three a day. Questions are posted to two staff members with one reference librarian as the 
lead responder or refen*er. The head of the reference depanment serves as backup in 
absence of a primary staff member and monitors the volume and nature of the questions 
to determine impact on overall work-flow. 

Staff training has not been a part of Morris County Library's electronic reference service. 
This is due to the fact that the librarian handling electronic reference has had a significant 
amount of experience in academic and public reference (14 years), electronic 
communications (moderated chat rooms and bulletin boards as AOL online volunteer for 
three years), and Internet training for teachers, librarians, and the public since 1993. 
During her experiences, she has become of aware of specific issues related to electronic 
reference and the skills necessary for dealing with these issues. The table below offers 
issues and skills based on the electronic reference done through April 1998. 



Table 3-6 Electronic Reference Service Issues and Required Skills for Morris County Library, N J. 



Event or Problem 


Service Issues 


Skills Required | 


1. Service reaches worldwide user 


Define patron universe: 


Staff must know how to 


base including users from India, 


• Who will or wonT 


• Read domain names 


Brazil, and Philippines. 


you serve? 

• To what extent will 
international users he 
served (c.g., short 
answer, referral, 
transmit materials, 
etc.)? 


(country end codes) of e- 
mail addresses and use 
WHO IS or convert domain 
to Web URL to identify 
source area of query. 

• Use tools like American 
Library Directors^ Special 
Collections, and LibWeh to 
locate remote patrons' 
nearest collections. 


2. Service receives vague, ill- 


Identify steps to lake in 


Staff must be able to recognize that 


worded query. 


initial respon.se to patron: 

• Individual follow-up 
or template? 


• Patrons often have no 
concept of the possibilities 
of information, so query is 




• Re-statement of query 
only or initial 
information 
provided? 


tentative. 

• Query may be from patron 
writing in a second 
language. 



■■ 7'his table outlines skills required in daily activities of one electronic reference service. These skills can 
be translated into instructional objectives for training information specialists of this service and others. 
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kvent or Problem 



Incoming question refers li') a 
subject not best served by library 
resources (e.g., social services, 
law. quality of life issues, etc.). 



[\iiron seems to expect entire 
answer to be Internet -based. 



.T. Patron seems to expect immediate 



6. Requested infomiaticm cannot he 
readily prodded clectriMiicallv. 



Service Issues 



Define service scope: 

• Serve as gateway to 
luher agency 

in rorniaiiun*.^ 

• Does an electronic 
presence carry N\ith it 
an imperative to 
help? 



^ Skills Required 



Staff should recognize when to 

• Refer queries or mediate 
acquisition of information. 

• Decline question politely 
and helpl'ully. 




Define procedures for 
amount and format of 
transmittal of inrormation, 



Slafl* should be able to 

• Pro\ ide available 
information 

• l:\plain possible additional 
resources or information 

• Offer suggestions and 
routines lor non-electronie 
materials acquisition 
(inlerlibrarv loan, local 
eolleeiitms or auencies) 



SeiAiec receives repeat query 
friun same patron. 



Devise tracking system for 
queries answered and 
respondent. 



Staff should be able to 

• Recognize that c-pairons 
send queries to several 
libraries or lose answers 
through mail errors. 

• C'tMiipose neutral response 
asking if material is required 



S. Ser\ ice receives repeat qucslitui 
from several patrons. 



Ideiuirv processes foi‘ 
pixn'icling in formation on 
common teq^ies: 

• Keep an PAQ I'ilc. 

• Write and point to a 
W’eb page. 

• l^repare and pnn ide 
an information 
packet. 



Staff should be able t(^ 

• Recognize repeat queries. 

• Consult with colleagues on 
de VC leaping responses to 
common queries. 
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Event or Problem 



0. Lack of closure to service- patron 
exchange. 



Service Issues 



Identify procedure for 
following up with 
exchanges: 

• I Icuv oflcn'.^ 

• By individual or 
tcmplalo mailing? 



Skills 'Required 



Staff should recognize 
characteristics of 

• rompiciion of exchange. 

• Patron satisfaction. 



1 0. Nature of queries cau.ses staff to 
grow impaiiem. 



Pi\i\ ide techniques to 
motivate staff and prevent 
burnout: 

• Rotate c- ref staff. 

• Balance electronic 
reference with lime 
spent with local 
patrons. 



Staff shield 

• Recognize signs (^f 
irritability and burnout. 

• Decline e-ref responsibility 
rather than provide hasty, 
impatient responses to 
patrons. 



[ I. Satisfied patron threatens to abu.se 
the service. 



Create policy regarding 
limits of service and 
explaining reasons for such 



Staff should recognize when Ick 

• Politely decline queries. 

• Reroute queries to more 
appropriate agencies or 
offices. 



12. Staff feel.s tinahle to answer 
question. 



Track and analyze 
unanswered qucrie.s 
to gauge service and 
collection. 

Develop response 
method or form that 
respects patrons’ 
que.stions but explains 
why institution nr 
.service cannot 
provide answer. 
.Suggest information 
and individual or 
agency that may help 
answer quest ion. 



Staff should he able to recognize 
when questions indicate 

• User’s misunderstanding of 
the service. 

• Deficiency in the colleciion 
or in stafftraininu. 
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Module 4 

Prototyping 

Creating and Testing AskA Services in a 
Controlled Environment 

Module Profile 

Once an AskA plan is created, the service should be tested in a controlled environment. 
Organizations that have launched services to the public too soon have experienced 
question overload, sometimes to the point of having to discontinue service. Prototypes 
allow organizations to expand gradually with the opportunity to adjust plans (including 
policy and procedures) based on initial feedback and experience. This module is designed 
to assist organizations in testing and developing an AskA service according to an AskA 
plan. At the conclusion of this module, you will be able to achieve the following goal: 

Goal Create a prototype and conduct a pilot test of an AskA service. 

Prerequisites Before prototyping an AskA service, an organization should be able to 

1. Collect information on the general digital reference field and 
existing AskA services (see Module 1: Informing). 

2. Determine the best way to build and maintain digital reference 
service within the organization (sec Module 2: Planning). 

3. Plan, produce, implement, and manage training programs for 
service staff and information specialists (sec Module 3: 
Training). Note; Services may implement the training program 
pilot test during the prototyping stage. 

Objectives At the completion of this module, you will be able to achieve the 

following objectives: 

1. Understand the puiposc of creating a prototype of an AskA 
service. 

2. Identify common factors in prototypes of AskA services. 

3. Describe impoilani components for pilot testing an AskA 
service prototype. 

4. Identify issues and questions to he addressed during a pilot test. 

5. Revise prototype based on pilot test results. 



Informing 

Planning 

Training 




[Prototyping 


Contributing 

Evaluating 
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4.1 Prototyping the AskA Service 

'Fhc step between planning and implementing an AskA service is a crucial one. The 
prototyping stage represents a period of incubation when the service functions wwh some 
of its planned components but in an atmosphere of formative testing — with the jn^ssibility 
of revision and expansion. Some organizations and individuals have learned the hard 
lesson ol’ pubiici/ing an AskA service before important features are tested and before an 
underlying structure is in place to support it. This has resulted in such an uncontrollable 
number of questions that a service is forced to eliminate its opportunity for gradual 
growth — and, in some cases, must discontinue altogether. Prototyping can help an 
organization plan for service growth to avoid such disasters. 

Given that many AskA services begin as experimental efforts, the prototype phase is noi 
always considered separate from the overall development process. Many services seem to 
begin small and grow naturally based on increased demand and resources. I’hosc services 
that begin with a prototype are able to try strategies in controlled environments (through 
trial and error) before offering service to the larger po]nilation. This section discusses the 
concept ol‘ prototyping as it relates to the development of AskA scrs iecs and offers 
suggestions and examples lor creating an AskA prototyjK' prior to conducting a pilot test. 
Although the Training phase is placed before Prototyping in this text, the development of 
a training program may take place after the prototyping stage when issues of policy and 
j)roeedurc arc more firmly established or may occur in eonjunetion with the prototyping 
stage (c.g., the training program pilot test may be implemented wiih the service prototype 
pilot test). 

Prototyping: (General Detiiiition.s and Concepts 

The practice of prototyping — the devclojunent of an incomplete hut usable \'crsion of a 
product — has been used in a variety of contexts including engineering, software 
dc\'clopmcnt. and manufacturing. Prototypes can assist developers in assessing a prodiiel 
by providing feedback without requiring the ewpense of a full development cycle (Jones 
VlcrrilL 1992). One approach to prototyping invol\-cs a process of quickly building and 
testing a prototype early on in the development process in order to make more focused 
decisions regarding devciopmenl of the final product. This is referred tt) as rapid 
prototyping, a practice that has become increasingly widespread in software 
development as a way to ensure that user needs arc adequately met bcl'ore the final 
product is in their hands (Mullin. 1990). [^apid prototyping is considered more flexible 
than traditional techniques and can be more cost efficient. 
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The concept of rapid prototyping works well in the context of building AskA services, 
because the nature of AskA service is one ofeonslant change. As pre\'i()Lisly mentioned, 
mosi services tend to start out on a smaller scale naturally: this is due to a variety of 
factors: low supply of resources and funding; question of future commitment on ihe part 
of the managing orgiini/ation; slow start to publicity; etc. As ihe scr\ ice grows, user 
questions and oihev defectors (see Module 2: Planning) will constantly inform the 
processes and the ultimate service. In this way, an AskA serx’ice is in a constant iteraliw 
state as opposed to a product that is dcvelo|)cd and ultimately sold in its final form. 

Prototyping AskA Services 

Although prototypes of AskA services are more abstract than those of software or other 
types of products, they can still ser\’c to provide organizations with important information 
before a service “goes live.” AskA prototypes involve manipulation of detectors, rules, 
and effectors to create a manageable set of factors. In other words, an AskA prototype is 
created by adapting the original AskA plan to tcmporai'ily run on a smaller scale. For 
instance, an AskA plan may indicate that a service will use 500 engineering experts I'rom 
all over the country ("Resources: numan Resources”) to answer questions of K-12 
students all over the world ("Fffectors” "). The AskA |)rolotype could include 10 graduate 
engineering students from Carnegie Mellon University with a user base of students from 
local Pittsburgh higli schools. 

When creating or establishing a prototype of an AskA ser\ iee, it is primarily impev.-tant to 
consider the manipulation of the AskA plan that is least expensive and time and labor 
intensive. However, the prototvpe must still represent the planned service adequately 
enough to provide useful inldrmation. 

Common Factors in AskA Prototypes 

Some AskA services ha\c deliberately created prototypes, while others began naturally 
on a smaller scale and developed over lime. New AskA services can learn lessons from 
both experiences in terms of original service structure and eventual growth. Services can 
use a combination of the following factors (and others) to ci'catc their AskA prototype. 



' While ;i service may hinc the means \o gather inlbrniaiiim on .students (l)eleelors: Users), the aelual 
service (in terms oT responses users) is considered an output (bn'oetors). 
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Controlled Group of Experts {In formation Specialists) 

One common method for creating a prototype is by controlling the number of experts 
who will answer user questions. By keeping this initial group small, service 
administrators can control time spent on training, question distribution, assessment of 
responses, and other management issues, while taking the opportunity to test the overall 
process. Academic environments are especially conducive to prototyping AskA services 
in that they can supply organizations with available sets of students, faculty, and staff 
eager to share their expertise with inquiring users. The following are examples of 
methods taken by existing AskA services to control their original expert groups: 

• Representatives from sponsoring higher education institution (students, faculty, 
staff, etc.). Ask Dr. Math originally launched its service in 1994 with a small expert 
base of students from Swarthmorc College, the institution housing the service and its 
parent project, the Math Forum. Today, Ask Dr. Math has over 225 volunteer “math 
doctors" from all over the world. (See case study, "The Evolution of Ask Dr. Math.”) 
The MAD Scientist Network, of Washington University Medical School, began in 
September 1995 with an expert base of 25 scientists (professors and graduate and 
medical students) from the medical school; this number quickly grew to 40 scientists 
by the next month. MAD Scientist administrators posted fliers at the medical school 
in order to recruit the scientists. MAD Scientist’s expert base has expanded to over 
600 scientists from around the w'orld. (See case study, “Planning an Ask-A-Scientist 
Service.”) 

• Ultimate expert pool, limited number of participants. KidsConneci, from the 
American Association of School Librarians, gradually grew its volunteer base 
through a deliberate pilot effort. Prior tc the service’s “go live" date, interested 
experts (library media specialists) responded to requests for volunteers posted on a 
professional listscrv, and the KidsConnect Coordinator organized a series of four pilot 
and training sessions. By the time the service went public, KidsConnect had 30 
volunteers (starting with an original group of 10); today the service has over 200 
volunteers. (Sec case study, “Pilot Testing the KidsConnect Service.”) 

• Existing service staff. AskERIC, of the Educational Resources Information Center, 
began in 1992 with two full-time staff members who responded to user questions and 
performed general service management tasks. Gradually the staff base expanded to 
over 50 educational information specialists. (See case study, “Managing Grow'ing 
Numbers of Questions for AskERIC.”) 



138 1 ■ 

a. 4 




Prototyping 



Module 4 



In addition to gradually increasing a service’s expert base, services should incrementally 
increase support staff (e.g., moderators, team leaders, etc.) over lime. AskA prototypes 
can include small numbers of support staff with plans to expand. 

Comrolled Group of Users 

Some services begin with a limited user base in order to control the number of questions 
coming into the system. Methods of limiting users can involve targeting communities that 
arc accessible to the organization. Like the controlled group of experts, this works 
especially well for academic communities where inquisitive students arc in abundance. 
Services have also found it possible to limit users to those in a particular geographic 
location. Some examples include the following: 

• Users from experts’ communities. Library media specialist volunteers from the 
KidsConncct prototype responded to questions of students in their own schools in 
order to practice the required tasks and simulate the question-answer process. 

• Users from service and organization community, The MAD Scientist Network 
originally created its Web site to allow' K-1 2 students from St. Louis public schools 
to ask science questions of faculty, staff, and students at Washington University (part 
of the “Young Scientist Program" medinfoM'ustiedu/'-ysp/. a student-run program at 
the medical school that aims to increase the science literacy of area K-12 students 
through hands-on learning). 

• Users from limited geographic area. AskERIC selected three state networks to post 
and highlight its .service. It chose NYSERNct in New York, SENDIT in Noith 
Dakota, and TENET in Texas, These state networks were the most developed and 
provided the greatest support to K-1 2 educators (AskERIC’s population) at the time. 

Mil t imal d\ 'crl is iug 

In the prototype stage, organizations should be extremely careful about publicizing their 
services. It is advised that services do not offer methods for capturing user questions 
other than those from pilot groups during the prototyping stage. It is important that 
advertising efforts be directed only tow'ards those users and participants W'ho have been 
identified in the prototype. 

Some services may choose not to publicize at all until the service is ready to go public. 
Others may choose to make potential users and participants aware of current efforts as a 
way of building enthusiasm and support for the service. Publicizing at this point is best 
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kept to professional forums where organizations can increase awareness of and 
enthusiasm for the upcoming available service; common methods include presenting and 
exhibiting at professional conferences and announcements on professional discussion 
groups. See Module 5: Contributing, for examples of publicizing active services. 

Available Technology 

Obviously it would be most favorable to select and implement a system that could be 
easily adapted to ‘live” service from the prototype. Since it is difficult to predict the 
changes in service structure as well as changes in technology, many services stall with 
one system and then migrate to another. (See Module 5: Contributing, for tips and 
suggestions on selecting or creating software for AskA services.) 

Below are some examples of technology solutions employed by AskA services during 
their prototyping stages: 

• Borrowed server space. Services may seek Web space from an external provider in 
order to get up and running. For instance, the National Museum of American Ail 
(NMAA) Reference Desk originally housed its site on America Online (AOL). This 
an*angcmenl worked well, because AOL was local to NMAA and the AskA service 
originally lacked funds for staff and equipment of its own. (See case study, “Getting 
Underway: The Virtual Reference Desk at the National Museum of American Art.”) 

• Limit user interface. Some existing services began by using e-mail only and 
eventually migrated to a Web site (on which they provided user query forms as well 
as information resources). Working in this mode during the prototyping stage helped 
these services to focus more on the question-answer process and policies and less on 
resource creation and dissemination. 

• Create original software. While clearly the most time consuming and talent 
dependent, this solution has been used by some services in order to automate specific 
service functions. During the prototyping stage, the soflw'are is built, tested, and 
revised. The MAD Scientist Network created its own PERL-based software package 
called Moderator, which was built in 1995 and has evolved as the service grew (Bry, 
1 997). 

There are some important factors in selecting software in the prototype phase. Software 
.selected should incorporate four aspects as highlighted in Table 4- 1 : 
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Tabic 4-1 Factors in Selecting Software for AskA Service Prototype 



OpL'fi: Open software allows for growth and Supports open 

change. It conforms to standards tliat arc developed standards 

in the "public’* space. By conforming to open 

standards, an tn ganization becomes less dc[iendcm 

on a single software provider that may ladictilly 

change discontinue a product. An c.xamplc of 

this would he propneiarv mail sn stems that ha\e 

either been disconiiiuicd or changed li) adai'it tt) 

tjpen Internet standards. This docs not imply that 

proprietary solutions arc inherent!) had (or e\cn 

more risky), hut rather scrscs as a caution when 

one is shopping tor software. Open software can 

also refer to an organization's ability to modify the 

pn^duet as needed. In man\ ca.scs .sidtware is 

proxided with source code, allow ing an iiuliudual 

or team to rewrite portions of the code. 

C’an add features or 
customization 



Example 




ABACI li: Web sever or (he 
support of the C’onimon 
Gateway Interface (CXH) for 
Web programming. 



Sidldhh-. Prototyping provides valuable* learning 
opportunities to AskA services. In manv ways, (he 
priitotvping period will be the longest time a 
service has to sv stematicallv think abtuit the .\sk.-\ 
.service. If this learnmg di>es not occur with the 
final software selection, it is an oppt^rUinity lost, 
'rherefore, a software package that ean he sealed 
up for implementation is preferable. So choose a 
system that can accomincidate final growth 
projections: thinking big early will .save [ime. 
monev, and effort later. 



C'an iimw as th 



service iirmvs 



Buy software tliat can license 
more users as the service 
grows; that way vou can pay 
Lis you grow. I 'or I’rce sol (ware 
look for pcrforniLince i.ssiics 
and multi- user capabilities. 



BEST COPY AVAILABLE 
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Facet 


Example « ' : I 


Supportable: The largest question when selecting 
software is who will lake care of the product after 
purchase. Support is the number one issue when 
making a buying or adoption decision. An 
organization can choose to pay for support, making 
bugs, fixes, and maintenance someone else’s 
problem: it can hire an expert to keep the software 
running; or it can look for an active support 
community that is willing to help out. 


Paid support 


Support contracts and toll-free 
telephone number support. 
Some software packages 
include this support in the 
price; others offer it for an 
additional cost. 


Internal support 


Finding software development 
and support expertise can be 
difficult and expensive. 
However, having someone on 
payroll (or a responsible 
volunteer) can provide the 
maximum support and 
customization ability. 


Community or user 
support 


This is how the Internet was 
founded, and there are still 
many products (such as the 
Linux operating system) that 
utilize a community of experts 
who are willing to volunteer 
the knowledge and assistance. 


Infernct enabled: Digital reference services aren't 
by definition Internet services. It just turns out that 
the Internet is the only game in town to link to 
communities beyond the walls of an organization, 
such as the K-12 community. The irony is that 
many help desk solutions you can buy don't fully 
support the Internet. Software selected must be 
able to take questions from the Internet (in the 
form of either e-mail or Web pages) and must be 
able to post the answers back to the Internet (once 
again with e-mail or the Web). 


Web 


Can the software take and 
return questions via the Web? 
Will answers be archived 
automatically in a Web- 
acccssible knowledge base? 
For many users the Web is a 
better alternative to ask and 
get questions than e-mail. 
Public library users, for 
example, may not have 
individual e-mail accounts. 


E-mail 


Can the software accept 
questions via e-mail? 



There is still a lot of development to be done in the area of AskA software. Standards and 
available “off the shelf’ solutions are rare (if not nonexistent). Most organizations will 
need to boiTow existing code. The outlook for software is, however, bright. The 
government and other organizations are hard at work developing software for this very 
puiposc. Also, as companies see the importance of retrieving, tracking, and answering 
questions using the Internet, more software options will become available. 
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4.2 Conducting the Pilot Test 



Once a prototype is built, it needs to be tested. An AskA plan and its sirippcd-dovvn 
prototype may look good on paper, but what about implementation? How will procedures 
be carried out? Are policies realistic? 



The pilot test helps to answer these questions by allowing services to implement the 
prototype in a controlled manner. In other words the service will run using a limited 
number of experts or users (or both) and incorporating any other factors. In conducting 
the pilot test, special attention is paid to feedback of participants in terms of making 
decisions for revision of the AskA plan. This section discusses activities involved in the 
prototyping stage in order to capture important feedback. 

Ingredients for Successful Pilot Test 



There are no h.ird and fast rules for conducting a pilot test of an AskA service. The actual 
events and components will depend on the service’s original plan and prototype. 

However, the two most important factors to consider in any AskA service pilot test are 
control of the overall testing effort and an open forum for communication between pilot 
panicipants and service administrators. 

Controlling the Testing Effort. While some services may loosely consider their 
beginning stage as a pilot, it is most helpful to plan some parameters by w'hich to conduct 
a formal pilot test effort. This can work by organizing a specific timeframe for the pilot to 
occur and setting an agenda. Just like any other type of evaluation, the pilot test should 
begin by identifying specific objectives to guide the effort. The service should consider 
what it wants to accomplish by conducting the pilot test. In other w’ords what are the 
major questions that the service hopes to answer in this test of service? Examples include 

• Will the software siippon the planned question-answer procedure? 

• Will the training program adequately prepare information specialists and staff for 
their roles? 

• Arc service policies feasible? 
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More specific questions for pilot testing arc listed in the section “Questions Asked 
During the Pilot Test/' 

Providing Opportunities for Communication. It is CKtrcincly imponani to allow an 
open line of communication between all participants and service administrators. While 
observation of events is helpful, it is participant feedback that can offer insight into the 
types of issues that could arise after a public launch of the service. For instance, are the 
expectations of volunteer experts realistic (c.g., time limits, required steps for composing 
responses)? Arc users satisfied with the types of responses offered? Are there any 
political issues that may cause individuals to decline participation as cxpcils or cause 
potential users to not lake advantage of the service? These and other issues can be raised 
and discussed with pilot participants before the service is launched. Communication 
should occur throughout the pilot, but it is also helpful to amange for a formal 
“debriefing” in order to summarize and resolve important issues. 

Methods for encouraging communication include the following: 

• Survey — by e-mail or Web to get initial reaction to ideas, concerns, etc. 

• Lislserv discussion — some standard questions, but encourage discussion among 
participants. Gives pailicipants sense of ownership in project — impoHant for 
volunteers who don't gel repaid in other ways. 

• r^ace-lo-face meetings or focus groups — if all participants arc onsite (or in close 
proximity). 

Questions Asked During the Pilot Test 

Although many of these issues will be raised in the planning stage (sec Module 2), the 
pilot lest may raise the following questions regarding question-answer policy and 
procedure: 

• What resources should experts use when responding to user questions? 

• What technical capabilities should be required of experts and users? 

• What should be the general nature of a response (c.g., factual answer vs. pointers to 
resources)? 

• What components arc necessary to include in a response? 

• What is the .sequence of events taken in responding to a user query? 

• How much lime docs the question-answer process take? 
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Other questions may relate to the comfort level of experts or information specialists in 
responding to user queries (these questions may be included in a survey or other type of 
correspondence): 



• What was the most difficult or frustrating part of the process? 

• What were some positive aspects of responding to user queries? 

• What do you wish you knew before answering these test questions? 

• What did you learn from participating in the process? 



In addition to questioning experts during the pilot test, it is also helpful to obtain 
feedback from paiticipating users. If users arc young children, it may be necessary to ask 
a teacher or parent for feedback. Questions may include the following: 

• Were you happy or unhappy w iih your response? 

• Did you receive the type of answer you expected? 

• What other type of information do you wish the expert included in the message? 

• Would you use the service again? 

4.3 Scaling Up 



Revision; Turning Pilot Results Into Action 

Once the pilot Lest has been conducted and results gathered and analyzed, the service may 
identify areas for revision and further development. Table 4-2 show s an c.xampic of a set 
of results and possible actions during revision. 



Table 4-2 Example of Pilot Test Results and Corresponding .Actions 



Results and Feedback Actions^ 



1 . Too many rules to remember in terms of 

required resptmse components. 

2. Takes longer ihan suggested time to 

ereato response. 

3. Confusion regarding the type of answer 

to pit) vide (factual answer vs. pointer 
resources). 

4. Some participants eneounlored enlieisin 

from peers: that viruiai experts were 
replacing school-based professionals and 
res(nirees. 

.3. Need additional inidrmation from users 
111 order to wnie appropriate response. 



1. Create checklist for information specialists to refer to 

when composing responses. 

2. Extend required turnaround lime (t\so school days as 

opposed to one school day). 

3. Rc'. ise policy regarding nature of response and goals of 

serxice (include sol of inhirmation restmreos in each 
response unless question requires simple answer). 

4. Change in policy to rcllect that all responses must include 

a rcrcronee back to the students' school resources (library 
media specialist, teachers, print and clcciixmie resources) 
us appropriate. 

5. Revise policy to allow vcduniccrs to respond to user io 

gather more deuiilcd information; include fields on query 
h^rm asking .students to include informatam along with 
c]ucstH>n (grade level, assignment, siihject. etc.). 



best copy available 
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Making the Transition From Prototype to Contributing Service I 

If pilot lest results indicate the need for large-scale change, the service may begin a new I 

prototyping phase and subsequent pilot test. Some services may conduct several pilot 

tests before they arc ready to offer the service to the public. The appropriate lime for j 

transition depends on the individual service’s resources, experience, and to some extent, 

willingness to take a risk. j 

In any case, the transition from prototype to ‘live" service should be a gradual one. As 

previously discussed. AskA services experience development and change on a continual I 

basis. After a successlul pilot test, the service can gradually add staff, users, and other 

.service components as resources and experiences allow. Once a service is in its next I 

“'Contributing" stage, it will have the opportunity for constant evaluation and revision 

overtime (see Module 6: Evaluating). I 



Module 4 Summary 

'fills .section pros ided information, suegestion.s, and examplc.s for creating a prototype of an AskA service 
and conducting a pilot test. 

• Proioi>pcs of AskA services arc rcprcscntatnms of the ultimate service that allow organizations to 
simulate service in a controlled environment while utilizing (he lea.st amt)unt of resources. Some 
services experience this stage as a natural starting point; (Mhers make a distinct effort to conduct a 
formal lest before the service is made public. 

• Organizations can manipulate AskA plans to create a prototype by: 

1. Controlling (he group of e.xperts or information specialists 

2 . Ct)nlrolling the user audience 

}. Limiting advertising to professional outlets {stressing awareness of service) 

4. Lhihzing available technology {rather than waiting for the perfect solution). 

• Pilot tests are eondueied to lest the prototype and .should 

1. Maintain control of time, activities, and objectives 

2. Provide opportunities for communication between all participants and service administrators. 

• Depending on pilot test objectives, questions and discussion can address qucstion*answer policy and 
procedure, comfort level of participants in conducting job functions, and u.ser satisfaction. 

• Pilot test results should lead naturally to revision of the proic^type. Revision activities may include 
rest>urce creaium and revision of pc'ilieies and procedures. 

• Using the revi.scd pnuotype. organizations can transition smoothly into active service, or the 
Contributing stage (see Module 5). 
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Pilot Teslinjj. the KidsConnect Service 

by Blythe Bennett 



About KidsConnect 

General Infonnation 

KidsConnect is a question answering, help, and relerral service for K-12 students on the 
Internet. KidsConnect is one of five components of the ICONnect Technology Initiative 
from the American Association of School Librarians (AASL), a division of the American 
Library Association (ALA). KidsConnect receives its funding from Microsoft 
Coiporation. The service is maintained by a KidsConnect coordinator who works at the 
Information Institute of Syracuse, the organization that also operates the AskERIC 
service. School library media specialists from around the world volunteer to assist 
students looking for resources for school projects and assignments. All volunteers 
participate in a training session facilitated by a KidsConnect trainer; trainees read the 
service manual and must successfully complete a series of practice questions before 
taking on actual student questions. 

History 

KidsConnect began in July 1995 as a project of AASL through their ICONnect initiative. 
A KidsConnect coordinator was hired in July 1995 to begin the initial planning stages. 
I'hc coordinator participated as an AskERIC question-answer specialist to learn the 
procedure of an Internet-based question-and-answer service. During the time of the 
coordinator’s owm training, a group of pilot test volunteers were enlisted and offered 
input as to procedure and policy of the service, and volunteer training materials were 
prepared. All this planning took place via e-mail among people who had not met each 
other in person (for the most part). Pilot testing began in November 1995, and there were 
four series of tests and debriefing sessions all conc^uctcd via e-mail. The pilot stages 
ended in March 1996, and the service launched in mid-April 1996 as part of the ALA 
Library Week activities. 



/ 
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Implementing^ the Pilot 1Y\st 

Purpose for Pilot Test 

The pilot testing was conducied for several reasons. First, it gave the volunteer media 
specialists the opportunity to practice the procedure for responding to student questions. 
These people were already skilled school librarians, but they needed the practice of doing 
their usual job with the e-mail interface rather than working face to face with their 
students. Second, the pilot sessions also allowed the KidsConnect coordinator to practice 
the sorting and routing system that would best fit the service and technological 
capabilities available. Also, some unexpected benefits surfaced. For example, some 
issues came up as a result of the pilot testing that were useful in making policy decisions 
that currently drive the project. 

Such decisions made during the pilot test included these: 

1 . KidsConnect will not ser\e as a fax or inieiiibrary loan service. 

2. KidsConnect responses will not pm\ idc direct answers to students but rather pointers 
to resources wiicrc an answer can be found. 

Pilot Test Ory^anizaiion 

Participants of the pilot sessions were solicited from the school media listserv, LM_NHT 
{littp://ericir. syr.edu/hujiei). Participants had to be school librarians who were also 
members ol' AASL. Ten volunteers in the Cnited States agreed to he a part of the first 
pilot session, the initial training and feedback team. It is interesting to note that of those 
Hrst lO people, nine are still w ith the project, three years later. These people conversed 
via c-mail for several months while the beginning stages ol'ihc training manual w'crc 
w’rittcn and tested by the coordinator. In November 1 995, this first group of lO volunteers 
all completed a series of responses to practice questions: the coordinator then provided 
feedback on each response. Formative feedback w as given after each practice question 
before any further t|uestioiis were sent so the \'oluntecr could leaim what was correctly 
done with a response and what needed to be improved. 

After this initial week of practice, there w as a debriefing session that allowed the 
\’olunteers to say what worked for them and w hat proved most difficult. The main 
drawback expressed was the time it took to respond to the questions. All the volunteers 
were in their ow n schools with their own jobs to conduct, so finding time to respond to 
qucslion.^ was the major problem they encountered. Prior to this pilot test, it w-as thought 
that volunteers would he rcc|ucsted to take no more than two questions a day, but this was 
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changed to no more than one questi* n per day, a policy that still is in elTcci. 

The second pilot lest session was much the same as the first, hut with a slightly larger 
group of pariieipants. h'ivc new participants were added to the original 10 to form the 
second pilot group of 15 jKopIe starling in January 1996. The third pilot test look place in 
hehruary 1996 with 20 participants, and the final test itx)k place in March 1996 with 30 
N'oluniecrs. Each pilot group built on the previous one, so the “veteran" volunteers 
became more comfortable and faster with composing their responses. Again, all 
participants communicated via e-mail, and most newer met in person but began to build a 
\’iriual community of volunteers. This aspect of community is a key component of 
toda>'s service; the \'oluntccrs feel a sense of belonging to a group working toward a 
common goal, although the relationships arc mostly virtual. 

Pilot Test Clialleni^es and Lessons Learned 

The main problem encountered during the pilot tests was the lact that the coordinator was 
using much faster technology than the paiiicipaiing library media specialists. Originally, 
ibc coordinator was not aw are of the longer search time required by school librarians in 
typical schools with lowcrend technology. Participants usually found it difficult to 
research and comptrse a response in the suggested 1 0- 1 5-minutc time period; on many 
occasions it took participants generally double the time. As volunteers got more 
experienced v\ith the way to respond and also w ith their own searching skills, they did 
take less time, but still not as short a lime as we hoped. One other area we could ha\c 
de\ eloped more before launching the .service was to set more policy issues: howcwcr, 
some issues didn't arise until the service was underway and the policies were established. 

Usin^ Results of Pilot Fest 

The service operates today much the way it was intended, but with some minor changes. 
Per instance, volunteers are not expected to answer more than one question per day, and 
training has been rel'incd and impro\ed over the first year based on feedback from the 
trainees. 

1'here were some unexpected benefits for the volunteers as a result of jxirticipating in the 
ser\ ice that arc interesting to note. For instance, volunteers ha\'c itulicalcd gaining a 
sense of accomplishment and intclleeiual stimulation, improvement of Internet and 
general reference and research skills, and the opportunity to make a positive impact on 
the local school community. 
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Tran sii ion lo Ruhlic S('rvic(' 

'Fhc iransiiion from pilot icsilng to public service was quite easy since there was no 
publicity and no large increase in the numbers ol’ incoming questions. I l(n\‘cvcr, as a 
word of caution, once you do plan for publicity, you need to remember to keep one nv 
two steps ahead of demand. Always keep addin; to your volunteer base if you want the 
scr\’ice to grow, maintain the turntiround time limits you set for the service, and he 
carelul to avoid burnout, especially if the service operates with volunteers as opposed to 
paid employees. 

StatT Motivation: Hnsuring ^Success Beyond the Prototype 

Any organi/.alion that relics on volunteer power must consider the benefits that 
volunteers can gain from paiticipation. Although they usually do their ])art of the service 
out o( a sense of professionalism or ioi fun, they need to benefit somehow from the many 
hours they put into the service. They have their own obligations at work and in their 
personal lives, so it is a true commitment to be a part ol' a volunteer organization that 
takes quite a bit of their time. They need lo reap rew’ards (whether expected or 
unexpected) in order to help keep their commitment and enthusiasm alive. KidsConnect 
can offer the following advice based on experience: 

• Offer positive and corrective feedback on volunteers' responses. It is im|HM’lanl lo let 
volunteers know that someone else has taken the lime to read and comment on their 
responses on occasion. 

• Hstablish a sense of a vimial community so that the vidunteers feel a part of a group. 
This helps build their confidence in serx icc participation as well as in their 
professional roles. 

• Help vt)lunlccrs transfer the skills and knowledge gained from the service lo their 
own job roles. 



Blythe Allison Bennett, h I vdi c (sP cri c • / r. s xr.edu . is the KidsC.'onnecl Q and A Coordinator. 
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operating and Maintaining the AskA Service 

Module Profile 



Once an AskA service has been icsied and rc\ iscd, an organization is ready to offer the 
service to its intended audience. This module is designed to prepare organizations to 
promote, manage, and maintain an AskA service. At the conclusion of this module, you 
will he able to aehieve the lollowins eoal; 



Coal 



Manage service dc\'elopmcm and operations tmd build 
partnerships to gain necessary support and share processes. 



Prerequisites Before contributing as an AskA service, an organization should be 
able to do the following: 

1. Collect information on the general digital reference field and 
existing AskA services (sec Module 1: Informing). 

2. Determine the best way to build and maintain digiltil reference 
service within the organization (sec Module 2: Planning). 

3. Plan, produce, implement, and manage training programs for 
service staff and information specialists (see Module 3: 
Training). 

4. Create a prototype and conduct a pilot test of an AskA service 
(see Module 4: Prototyping). 

Objectives At the completion of this module, you will he able to achieve the 

following objcctiN'cs: 

1. Publicize ser\ice to potential u‘ .v 

2. Identify strategics for handling increase in questions. 

3. Create resources to suppoil AskA service. 

4. Build partnerships with other organizations to support scrxiec. 

5. Recruit expert (information specialist) base. 
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5.1 Contributing an AskA Service: Putting It All Together 

The ongoing stage ofolTcring service to the K-1 2 community is refeiTcd to as 
‘ contributing.” A new service can contribute to the overall collection of AskA scrx iccs 
for the K-1 2 community in terms of incoming questions, expert responses, know ledge, 
and experience. In addition, the AskA service can contribute to its supporting 
organization by providing a mechanism for user feedback. 

5.2 Developing and Managing Service 

A contributing AskA service requires ongoing maintenance and troubleshooting and a 
constant eye tow'ards rc\ ision and change. 'Fhi.s section offers suggestions and examples 
for publicizing an AskA service to users, managing growing numbers of questions, and 
building suppoiling resources. 

Publicity Strategies 

Besides the obvious reason for publicizing an AskA scr\iec (i.c,, to gain users), services 
should also consider publicizing as a responsibility. AskA services providing information 
to the K-1 2 community arc responsible for informing potential users of the value that can 
be gained from use of the service. By offering this important information to the public, an 
AskA service makes an effort to reach K-I2 communidcs across the country and helps to 
decrease the gap between the “ha\cs” and the ‘'have-nots” in terms of effective learning 
opportunities. 

The extent and nature of publicity depends on the resources available to the organization. 
For instance, an AskA service with a large budget may consider launching a mail 
campaign to a large audience (c.g., to a mailing list of a professional publication or 
association) in order to reach poicnlial users who might not be as likely to come across 
the service on- the Internet. Sciwiccs w ith smaller budgets may choose to simply register 
with Web search engines and negotiate with related Web sites for placement of links to 
the service's Web page. 

A well-defined public relations plan can ensure that services arc well-publicized and 
promoted on a regular basis. It is common practice for new services or scr\ iccs with 
limited resources to avoid publicizing opportunities for fear ol' increasing their numbers 
of questions beyond control. This is a \ alid concern. How'cver, the overall publicizing 
plan should account for a gradual growth in numbers and resources over time. By making 
an effort to start slow ly in the prototyping stage (by targeting a limited audience), the 
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siLU’c IS alrcad\ set for a steady growth in publicity as well as scr\icc dc\’clopment, 
Methods for publicizing a digital reference service may include the following: 



Pniu-hused and Face- fo- Face Publicity 

• Target direct mail campaigns to potential users — A snaiUinailcd postcard or flyer 
works well to introduce the AskA service to even a non-lntcimct-savvy audience — 
and may c\cn provide them with a reason to get connected. It also guarantees service 
visibility among more than one audience even when the computer is turned off (c.g., 
postcards may travel around a school from teacher to teacher, be posted on a bulletin 
board, or be copied and placed in mailboxes). Mailing lists may be acquired from 
joumal subscription lists, professional association membership lists, etc. However, 
this possibility is realistic only for serv ices with large enough budgets to cove: die 
printing and mailing costs. 



• Distribute fliers and handouts — A less expensive option than direct mail, 
distributing paper-based advertisements by hand can increase the visibility of an 
AskA service. This possibility is appropriate for libraries and other institutions that 
experience ^‘drop-ins" to a central facility or for services whose reprcsciuativcs attend 
conferences and meetings. 

• Advertise in professional or institutional publications^ — Depending on the nature 
of the organization supponing the AskA service (as well as the resources), it may be 
appropriate to advertise in a professional or insiiiuiional publication (c.g., Journals, 
newsletters, etc.). 



• Attend and present at professional conferences and meetings — Many AskA 
services arc well promoted at professional conferences and meetings when 
rcprescnialivcs have the opportunity to present and explain imponanl information 
about the service and potential users and pariicipanis can ask qucsiioius. Besides 
presentation and discussion oppoilunilics, services can also present at trade show 
booths as a way to capture walk-through traffic. 




153 



Contributing 



Module 5 



lute met -based Publicity 

• Post promotional messages to appropriate electronic discussion groups and 
listservs — This is a good way lo target audiences with specific interests (c.g., 
educational technology, math education, parenting issues, etc.). A posting can travel 
from one list member's e-mail account lo that of a colleague. This is an inexpensive 
way (only cost is Internet connection) to advertise to a targeted audience (see Dr. 
Math case study). 

• Prov ide references to the service on related Web sites — This requires an effort on 
the part of the AskA service staff lo identify related sites and ask site publishers 
(Webmasters, etc.) to provide a link lo the AskA service. Depending on the nature of 
the airangement, this could entail a link or some information about the service 
accompanied by a link. For instance, the Virtual Reference Desk AskA+ Locator 
hnp://\\'\v\v.vnior^Aocator/ \)vo\’\dts links to and profiles of K-12-related AskA 
services that meet general quality criteria (see Module 1: Informing). 

• Link from Internet service provider pages — Besides sites on the World Wide Web, 
services may also consider an'angcments with Internet service providers whereby an 
AskA service is linked from a directory on a page accessed by members only. For 
instance, the National Museum of American An Reference Desk has a link from 
America Online’s “Reference and Learn Channel.” This allows the service to attract 
people whose central (or only) point of access to the World Wide Web is through the 
provider (sec National Museum of American Art case study). 

• Palace reference to AskA service prominently on own organization’s Web site — 

Organizations can control the amount of exposure of their AskA service by the 
placement of service access on their Web site. The highest level of visibility of a 
service can be gained through a link at the “top” of the sponsoring Web site (i.e., the 
home page or one of the first Web pages accessed upon arrival to the site). For 
instance, AskERIC experimented by placing a colorful “banner" graphic on its home 
page reading, “Got an education question? AskERIC," which linked lo the service’s 
question subinission form. This nearly tripled AskERIC.’s question load (see 
AskERIC case study). 

• List site with search engine — An organization can register with major search 
engines (c.g., Yahoo!, Alta Vista, WcbCravvlcr, InfoScck, etc.) in order to ensure 
prominent placement of a Web page in users' search results. Completing a 
registration form for a panicular search engine allow s the AskA service 
representative to present key information about the AskA service in an organized 
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manner. (For meta-guides to search engine registration, see SitcOwner 
htlp://\v\v\\\sUeo\\iier.com/ ox Submitit hup://\v\v\v. submit-it.com/.) However, it is not 
necessary to register with a search engine in order for a particular site to appear 
during a search. Search engines pick up Web pages based on keywords. Therefore, it 
is important to include keywords on an AskA service's Web pages that reflect 
common questions or topics. 

Managing Growing Numbers of Questions 

If publicity attempts are successful (whether intended or not) and if your service is of 
value, users will catch on. As more people and classrooms become connected to the 
Internet. AskA services experience more and more traffic. AskERIC has seen an increase 
from 151 questions per week in 1992 to 868 questions per week in 1998 (sec AskERIC 
case study). KidsConnect experienced 1000% growth during the month of September 
I996 (Lankes, 1997). 

It is projected that by the year 2000, there will be approximately 54 million students 
enrolled in elementary and secondary schools in the U.S.: 39 million in grades K-8 and 
15 million in grades 9-12 (Lankes, Bry & Whitehead, 1996). The implications for AskA 
services may seern frightening, but services can prepare themselves for high volume 
using certain strategics and suppon mechanisms. The first step is to make a gradual 
transition from the prototype phase to the contributing phase (see Module 4: 

Prototyping). Below arc some additional strategics and support mechanisms for ongoing 
control of question volume (also see AskERIC case study): 

1, Gradually increase staff and expert base. Whether a service functions with full-time 
paid employees or part-time volunteer experts, most services have learned that increasing 
human resources is crucial in terms of managing increasing user questions. In order to 
address the greatest number of incoming questions, AskA services require a team of 
individuals dedicated to answering questions and supponing the work of those who 
answer questions. This team can be built over time according to the resources available 
and processes in place. For instance, a service may be able to add only a small number of 
new experts at one time, because each new participant must undergo a time-consuming 
training process managed by a limited number of staff members. 

2. Automate processes as needed. Over time (or sometimes right away), many services 
discover that some manual tasks can be completed more quickly with the aid of a 
computer-based system. This can range from distributing a stock e-mail response 
informing a user that his or her or her question has been received to running an incoming 
question through a knowledge base to determine if a response is already in the system. 
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Another method of automation is to provide Internet-based resources to supplement the 
question-answer process (e.g., archive of questions and answers and lists of frequently 
asked questions; see Building Information Resources), Below are examples of original 
software packages developed by Ask A services to automate their specific tasks: 

♦ Doctor^s Office — This software package written by "Math Doctor” Ken Williams 
lists incoming questions in a "triage area” in which information specialists select 
questions to answer and place answered questions and responses in a "post-op area” 
where responses can be checked by other staff before going out to users. The software 
also accommodates administrative tasks and access to resources and includes a help 
function {iittp://fonuii.s\vartliuiorc.eclu/dr.}nath/officeJ}elp/dr.()jJiceJielp.lUfnl). 

• MODERATOR — -This software from the MAD Scientist Network docs several 
functions including formatting incoming questions to HTML; adding and modifying 
data in the databases of expeits, etc.; maintaining statistics on expert performance; 
reviewing questions and answers; tracking routing of questions; and general site 
maintenance [hup://\v\v\vjnadscior^/ask_expert/2ditnil), 

Hou'ever, no matter how much automation is implemented, it is important to maintain a 
level of human intermediation. The value of AskA services is due in large part to the 
quality of person-to-person communication. 

3. Maintain standard policies for recurring procedures and issues. Policies for 
question answering and other procedures not only ensure consistency and quality in 
responses to users (and other service resources), but help to speed up processes as well. 
For instance, if an expen is required to include a set of components in every response, the 
act of including these components in a response becomes a habit for the expcil. Policies 
are created in the planning stage and are communicated to experts and staff during 
training; however, it is important to constantly reinforce policies through mechanisms 
such as manuals and job aids for use after formal training programs. 

4. Control publicity. As previously mentioned, publicity efforts should be handled 
carefully so as not lo generate more traffic than can be accommodated at a given time. 
Major publicity efforts such as large-scale mailings or prominent links on popular Web 
pages should be implemented only when staff (experts and suppoii crew) and question- 
answer processes are prepared to address each question according to the AskA service's 
policies and goals. 

5. Share questions and knowledge. AskA services can partner vsiih other services and 
organizations to ease question loads and facilitate the question-answer process. Out-of- 
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scope questions can be forwarded to more appropriate services, and responses generated 
from one AskA service can help information specialists from other ser\ iccs compose 
responses. The Virtual Reference Desk Project provides a set of resources and services to 
promote this type of question and knowledge sharing. The technical infrastructure allows 
AskA services to forward oui-of~scopc or o\crload questions to the central Virtual 
Reference Desk location to then be distributed to other services. (Sec Building 
Partnerships section.) 

Building Information Resources 

Many AskA ser\’ices view resource de\e)opmcnt as a significiint component of the 
seiwice: others dc\ote loss time and energy to resources and more to the ciuestion- 
answering process. As with most decisions regarding AskA ser\'iccs, this one depends on 
time, funding, and staff of the service. Some AskA service information resources require 
a large amount of technical talent and financial resources, while others require basic 
HTML skills and access to a Web server (c.g.. through sponsoring university). 

Resources such as an archive of questions and answers, lists of frequently asked 
questions, and supplemental information serve both to facilitate the question-answer 
process and to reinforce important subjccl information for K-12 users. Decisions 
regarding information to be included on a service's Web site usually stem from feedback 
in the form, of user questions and other communications. In building AskA service 
information resources, organizations should consider content, format, technical, and Icual 
and ethical issues. Below are some suggestions and c.xamplcs to help organizations build 
information resources for their AskA services. 

Building and Or^anizin^ Coiucni 

As previously mentioned, AskA service Web site content is often based on user needs as 
identified by user questions and other forms of feedback. For a beginning service, content 
can reflect the general scope of the service and identified needs of the potential audience. 
When considering content to include, AskA service representatives should decide what 
information is most useful to the user and most accurately rcnccls the .service scope. It is 
also helpful to consider what information would be useful for information specialists or 
c.xpcrts in creating responses. 
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Organizations may want to include information that 

• Dissuades users from submitting commonly asked, basic questions to the service 

• Introduces users to the general subject matter 

• Draws users back to the site, or 

• Rctlects in-scope questions (e.g., actual questions or examples of the types of 
questions the service would like to receive). 

In addition to choosing relevant and helpful information, service.^ should ensure that the 
information is easy for users to access. Below are some common types of information 
found on AskA service (and supporting organization) Web sites and examples of how 
information is presented: 

Prevlou.sly-Asked Questions and Answers. A large number of AskA services provide 
access to some or all user questions and corresponding responses. This type of collection 
is commonly referred to as an archive. Archives can be organized by topic, date answered 
(month and year), grade level or age of user, etc. Question-answer sets can be accessed 
through link indexes, search engines, or both. 

Examples of searchable archives: 

• MAD Scientist Network http://madsci.\\'usiledii/MS_search.html — This archive is 
searchable by keyword. Users can select options for Boolean searching, files to 
search, hits per page science area, and grade level. Results appear as lists of 
questions, representing either the original question or a response, accompanied by 
information about the person who posted the message (i.e., either the user who posted 
the question or the scientist who answered it). Users click on the question to link to 
the full-text question (which contains a link to the response) or directly to the 
response. (The search interface runs with Apache’s mod_perl; the search script is a 
front end to a Glimpse server.) 

• ScienceNet Database http://\v\v\v.scicucc}ict.or^Aik/cjpages/scarch.ht}}d — The main 
search page allows users to select a section of the question-answer database to search 
(e.g., archaeology, chemistry, etc.). An option is also available to search the entire 
database. The database uses a Glimpse search engine. 

• How Things Work 

http://}a}idanl .phys.yir^hua.edu/Education^rcachin^/now'TIdm^sWork/qsearchditnif 
— Users can search for keywords in questions or answxrs. Results appear as a string 
of full-text questions and answers. 

When including a search engine on an AskA service site, it is a good idea to include a 
“help" section with directions for the user audience. 
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Examples of listed or categori7,cd archive: 

• Ask Dr. Math hup://f()mm.s\\'arihmon\echi/ch\maih / — In addition to offering a 
search engine for its archive, Ask Dr. Math provides access to its questions and 
answers through an organized set of links. Users may choose a section of the archive 
based on grade level (e.g., elementary, middle school, etc.) and then topic. Links from 
the topic list lead to full-text question-answer sets. 

• Ask A NASA Scientist 

hup: //imagine. gsfc.nasa.gov/docs/ask_astro/ask_an_astronomer.huul — Th i s si tc 
organizes its question-answer sets by topic but highlights those topics that reflect the 
scientists’ main areas of expertise (e.g., cosmology, quasars, etc.). Thus, the topic list 
on the main page is divided into two categories, cxpeilise areas and additional topics, 
no doubt in an attempt to encourage in-scope questions while not discouraging others. 
Topic lists link to question lists, which link to full-text question-answer sets. 

Frequently Asked Questions and Answers. Frequently asked questions, or FAQs, are 
also a common feature on AskA service Web sites. The questions to highlight in an FAQ 
may be determined in a number of ways; this can include automated tracking of incoming 
question topics or observation by AskA service staff w'ho arc familiar with incoming 
questions. Most FAQs are categorized by topic and sometimes further broken down by 
subtopic. By providing FAQs, services have the opportunity to highlight exemplary, 
sometimes comiprehensive, responses to • ommon questions. 

Examples of frequently asked questions and answers: 

• Top 101 Ask a Volcanologist Questions 

hUp://volcano.und.}wdak.edu/v\vdoc.^/frequcnijiiucstions/iop_WIfr op _1 01 .hard — 
This service requires that users review the FAQs in order to gain access to the 
question submission form (or any information regarding sending a question). This 
forces a user to make sure that his or her question has not yet been answered so that 
the small team of experts can control the number of questions received. 

• Internet Public Library Frequently Asked Reference Questions (FARQ) 

!itip://\v\v\\\ipl.org/ref/QUE/FARQ/FARQJ\tml — ^This FARQ page presents each topic 
bi‘okcn down into subjects; each subject name (e.g., literary criticism, genealogy) is 
accompanied by a sample question that can be answered by reviewing the particular 
FARQ. Since the Internet Public Library’s AskA service offers expertise in general 
[•cfcrcncc, the actual FARQ pages include some introductory information and 
rcicrcnccs to other resources. 
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Supplemental Information. Besides offering access to question-answer sets (whether 
from archive or FAQs), some AskA services provide additional information and 
resources to supplement the knowledge generated by the question-answer process. This 
can include references to related sources (e.g., links to Web sites), Web pages with 
factual information on related topics, or complete collections of related resources for the 
target audience. 

Examples of supplemental information resources from AskA services: 

• Ask Shamu littp://\\'\\'\\\sca\\'orhior^/ask_sliamn/asi}idcx.lit}ul — The responses in the 
question-answer archive include links to Web pages with detailed inl'ormalion and 
photographs on particular marine animals provided by the sponsoring organization 
(Sea World/Busch Gardens). For example, detailed information on killer whales is 
referenced in a response to the question "Can an orca whale have twins'?'* 

• KidsConnect Favorite Web Sites for K-12 Students 

hup://\v\v\\\ahi.or^/lCONN/kcfciyoritesJu}}\l — Volunteer library media specialists 
from the KidsConnect services created a list of Web sites relating to topics commonly 
addressed in student questions (e.g., animals, geography). The list of sites was 
intended to aid students as well as volunteers answering questions. 

• AskERIC Virtual Library hitp://m\'\\\askcn'iL\or^/\^iniud / — AskERIC has built a 
comprehensive set of educational resources for educators and others interested in 
educational research and practice. Resources in the Virtual Library include 
pathfinders to information on educational topics; lesson plans; educational listscrv 
archives; links to education-related Web sites; and links to other resources from the 
parent organization (Educational Resources Information Center). 

AskA services constantly strive to provide better access to information to their users. For 
instance, Ask Dr. .Math is currently looking into the possibility of automatically matching 
incoming questions with responses in the archive so that users arc directed to relevant 
information before the question is further addressed by an expert or service administrator 
(sec Dr. Math case study. Module 2: Planning). 

Le^cd and Ethical Issues and Solutions 

As discussed in Module 1: Informing, there are some legal and ethical issues to consider 
when providing an AskA service. These issues come into play not only in providing 
responses to users but in providing resources on the AskA service Web site as well. The 
rollou’ing discussion identifies legal and ethical issues along with solutions implemented 
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by some AskA services. In addition to the suggestions heiovv, services may also consider 
obtaining legal advice regarding these issues. 

L'ser Privacy and Confidentiality. When posting question-ansv\’cr sets on a public Web 
site (c.g., in an archive), AskA services should he careful not to provide access to 
inforination that can identify individual users in any way (sec Module 1: Informing). 
Options for maintaining confidentiality of user information include these: 

• Stripping all identifiers from messages before posting them on the site. For 

services with small numbers of incoming questions, this may be done manually (i.c., 
by deleting contact and other personal information IVoin the message); for those with 
larger question numbers, identilVing information may be stripped automaticaliy (i.c., 
by programming the system to remove information in certain fields such as “name," 
“e-mail address," etc.). 

• Limiting access to user information. While stripping personal information from 
given fields works in many cases, it is impossible to control personal information that 
appears in the body of users’ messages. Some AskA services do not trust technology 
to remove identifying user information from all areas of user messages. In this case, if 
manual revision is not possible, the service may want to limit access to the question- 
answer archiv'c to those with authority — AskA service staff and information 
specialists. 

Many services include user information in their archi\c, but make efforts to discourage 
the use of such information for commercial purposes (c.g., mass e-mail, etc.; see MAD 
Scientist disclaimer liilp://\\'\v\\\ifia(lsci.()r{i/i)}f()/clisclainierJu})i!). 

One way to ensure privacy during the question-answer process is to post user responses 
on password-protected Web pages. 

Copyright. As in any content de\cIopincnt, information on AskA service Web sites 
should be presented accurately, giving appropriate credit to original authors. Indi\'iduals 
developing content other than that based on information specialist c.xpcriisc shtnild he 
sure to cite when appropriate. 

.Another copyright issue involves services' ownership rights to information on their own 
sites (c.g., question-answer sets, FAQs, etc.). Many services may not own rights to their 
resources depending on individual service issues such as policies of sponsoring 
organizations (sec Module I ). Those that do wish to protect ihcir information from being 
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used w iihoLit permission (e,g., images or text downloaded for use on others’ Web sites) 
should include a policy statement on the ser\'ice Web site. 

Liability, This issue applies to all AskA ser\ ices to some degree but is especially 
important for services whose expert information can be interpreted as professional advice 
— medical, legal, etc. Protocol for guarding against liability in infonnation specialists’ 
responses should be addressed in the AskA service policy. (Solutions for communicating 
liability matters in responses can include creating an electronic signature with a 
disclaimer.) However, attention should also be paid to liability issues when creating 
resources for the AskA service Web site. Protecting a service against liability can be as 
simple as placing a disclaimer on the scr\icc's home page or question submission page. 

Examples of disclaimers on AskA service Web sites include these: 

• Ask die Optometrist liiip://\v\\'\\'.visi()ficarc\com/askJ\mil — The main page for this 
AskA service includes a lengthy disclaimer stating that by submitting a question, the 
user agrees that he or she docs not become the doctor’s patient, should seek care from 
a competent professional, and docs not hold the doctor or organization liable for care. 

• MAD Scientist Network }itip://\v\v\vj}\culscix)r^/info/disckii}ner.ht)nl — This site 
includes a disclaimer including statements that the information on the site (as well as 
in responses to users) docs not represent the views of the sponsoring institution or the 
institutions of individual experts; is for educational puiposcs only and should not be 
interpreted as medical or professional ad\ icc; and is not guaranteed to be completely 
accurate. (Instructions arc provided for submitting coiTcctions to errors found on the 
site). 

'I'cchnical Support and Devclopmeui 

Besides selecting and creating content for the AskA service site, a service needs to plan 
for the technical development and upkeep of the resources. This can range from creating 
simple HTML files of FAQs to building a search engine for question-answer archives. 
(Some systems integrate resource development and access capabilities with overall 
automation of the question-answer process such as question routing, etc. Sec “Managing 
Growing Numbers of Questions.”) 

For a detailed guide to software and systems for AskA services, refer to the MAD 
Scientist Network's “Setting Up an Ask-An-Expcri Service,” Section VI 
liiip://\v\v\\\uuulsciA)r^/askj:’.\pcr(/6^,l\ir}]l (Bry, 1997). Descriptions and suggestions are 
pro\ ided for Web servers, operating systems, mail programs, search engines, and 
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interactive communications tools for service administrators and information specialists. 
Information on product quality and cost is included. 

5.3 Building Partnerships 

I 

Individual K-12 AskA services are not alone in their goal to provide quali'y information 
I to ihc education community. AskA services, businesses, professional associations, 

government agencies, educational institutions, and other bodies can work together to 
j support various facets of AskA service: software, Web space, expert (information 

specialist) recruiting, publicizing, etc. AskA services can seek support through the Virtual 
Reference Desk Project as well as other associations promoting similar interests and 
goals. 

I he Virtual Reference Desk Project 

The Virtual Reference Desk Project is developing a set of resources and services to assist 
AskA services in managing their question-answer process. Specific components of the 
Project include the following: 

1. AskA Consortium. The AskA Consortium is a network of organizations in areas of 
technology, digital reference, library science, business, and education to help guide the 
Virtual Reference Desk Project and develop necessary resources and support. Included in 
the AskA Consortium arc AskA services that agree to adhere to quality criteria as 
identified by the Virtual Reference Desk (sec Module I: Informing). AskA services in the 
Consortium will participate in a process of question sharing and contribute question- 
answer sets to the Knowledge Base (sec below). 

2. Meta-Triage Function. The meta-triage is the central switchboard for questions and 
answers traversing the Virtual Reference Desk's network of users and knowledge 
resources. The meta-triage function takes questions from the K-12 community directly us 
well as from the AskA services in the AskA Consortium. In this way if an AskA service 
receives too many questions, overload questions can be routed to other AskA services 
that are underutilized. An AskA service can also send a question to the meta-triage 
function if it docs not match the expertise provided by that AskA service. In this way if 
an AskA service receives a question from a user and is unable to answer it (a science 
question sent to a math service, for example), it can use the meta-triage to get the user's 
question to the best knowledge resource. 

3. Knowledge Base. The Knowledge Base represents a structured collection of existing 

^ question-answer sets. It is distributed with its contents managed by the varying AskA 
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services. The knowledge base can help AskA services manage an increase in questions by 
providing a resource for users to consult prior to question submission and can help 
experts by providing information that can be used to compose responses. While 
individual AskA ser\'ices may have their own knowledge bases (or “archis'e’"), the Virtual 
Reference Desk Knovs lcdgc Base pros’ides a more comprehensive set of resources from 
all AskA services in the Consortium. 

Menibership in A.ssociation.s 

Partnering with professional associations can ofj’er many benefits to AskA services. Some 
services have in fact grown out of professional associations (c.g., KidsConnect is part of 
an initiative of the American Association of School Librarians [AASL], a division of the 
American Library Association jALAl). d'hese .services are more likely to start out with 
built-in expert and user ba.scs. 

There arc different models of working with a professional association: 

Association ownership, in this model the as.sociation owns and operates the service. J 

While outside contractors or experts may aid in the logistics of the service, it is the q 

associali(.)iTs ownershij) that drives the ser\ice. Associations can use an AskA ser\'icc to J 

provide increased training and de\'clopment opix)rtunitics to its members, increase I 

\ isihihl)' inside or outside its community, create a direct method for taking information 
from its users or community (through the questions they ask), or promote an agenda to a I 

larger community (such as the importance td' topic X in the K-12 curriculum). Sc\'cra! 

AskA services were created to fill percci\cd gaps in K-1 2 curricula (c.g.. art education). | 

AskA\ ser\ iccs benefit from such arrangements when associations allow the scr\icc 

acccs:; to user communities that httve already been identified. Por instance KidsConnect I 

promotion could coincide with AASL's promotion of reading in the schools and ALa\'s " 

promotion of public library use. In this case, the association provided access to its own 

users and its target populations ihi’ough traditional promotion techniques such as mailers. | 

In addition, associations can provide links to their jiartncr AskA scr\ ices from their home 

page, which could boost question numbers from appropriate u.scrs. I 

Association expert base. In this model the /\skA scr\ icc is run by a service or I 

oraani/ation external to an association. The association and externa! oraani/.ationa! 

mission arc at the very least aligned, but the association does not take on the longer term ■ 

1 unding and oj’)craiional ctmeerns. d’he association instead promotes the AskA service * 

internally to its members as a was* of seeking volunteers to work with the external AskA 

service, 'fhe as.sociation gains outreach and achiewes mission objectives without the I 

burden of AskA service support and fundiim. Members of professional associations often 

I 
I 
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reflect the most dedicated and talented representatives of a particular field. By offering its 
membership base as a possible information specialist pool, an association not only 
provides the K-12 community with needed c.xpertise but also offers meaningful voiimiecr 
and professional development experiences for its incmhers. Benefits of participating in 
digital reference services include improvement of general skills (c,g., Internet searching, 
subject knowledge) and personal satisfaction of helping users solve information 
problems. 

Association sponsorship. This model requires the least commitment and rc.sourccs by an 
association. The association hccomcs a promoter or resource provider to an AskA service 
without direct invol\cment in the day-to-day operations. The association acts as a sort of 
“scal-of-approNxil” agency to the AskA scia icc. 

While these models reflect how associations can become involved in AskA services 
(particularly K-12 AskA services), they do not dii'cctly address wh\* associations should 
consider becoming invohed. 

Such a discussion must ulliinalcly align to the mission and membership of the 
organization, hut the following is an initial list of reasons professional associations 
should consider creating, participating in, or sponsoring an AskA scr\acc: 

• Membership outreach. Association can use the AskA scn icc as a means to increase 
participation by its members. Answering questions requires less in\'ol\cmcnl than 
committee work and can be infinitely more rewarding. 

• Membership training. AskA services can pro\ idc a concrete and directed 
oppominity to increase the skill sets of an association’s membership. This training 
can be in topical areas asscK'iatcd with the organization, Internet skills, 
communications skills, etc. (see Module 3; Training). F'or example if members arc 
going to be answering questions on topic x, they may be expected to ha\c a certain 
IcN'cl of understanding of that topic. This may lead to the creation of standards and 
ccnification processes in this topic. 

• Increased attention to assotiation i.ssues by a given community. Associations b\ 
definition have an agenda. By providing an easy-to-use and credible sci\ ice to a 
targeted eommunily rather than static resources (i.c., Web pages), an association can 
become a prcfcired source of information (mi a given topic. As a preferred source, the 
as.sociation can further its agenda while meeting people's information needs. 

• increased visii)ility of the association. B\ answering the questions of a larger 
community, such as the K-12 cdueation community, an asst)ciation can send its 
message and create an identity beyond its membership. 



165 



Contributing 



Module 5 



• Recruitment of members. With increased visibility and participation opportunities, 
association membership can increase. For example, AskA services can send a short 
invitation to join an organization with every answer, 

• Highlighting association products and services. Answers given will most likely 
come from ready and reliable sources. By pointing users to existing association 
publications, sales of association products can increase. 

Professional associations have a great deal to gain in the creation of AskA services. They 
can use the Internet to ha\'C greater access to user and member communities than ever 
before. Through participation in AskA services, association members can promote 
impoilant information to the public, improve skills and knowledge, and experience 
personal rewards that come from satisfying the information needs of the K-12 community 
and other Internet users. 
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Module 5 Summary 

This section provided information, sueizestions, and examples I'or managing AskA service development and 

operations and building partnerships. 

• Organizations can develop and manage an AskA service through publicizing service to users. 

controlling growing numbers of questions, and building suppt^rilng information resources. 

1. Publicizing strategies for AskA services include print-based. face-t{)-face, and Internei-ba.sed 
methods to inform potential users of benefits gained from using the .service. 

2. It is possible to manage growing numbers of questions by gradually increasing stall and 
information specialist base, automating processes, maintaining standard pt)licies. controlling 
publicity, and sharing questions and knowledge with other services and organizations. 

3. Issues in building supporting information re.sourccs irtr AskA services include building and 
organizing content (e.g., archive, frequently asked questions, supplementary information): 
considering legal and ethical issues in providing access to information (e.g.. privacy, copyright, 
liability); and planning for technical development and support. 

• AskA services can build partnerships with organizations, associations, and other .services in order to 

balance question loads, recruit information specialists, and build user ba.sc. 

1. The \'irtual Reference Desk Project provides resources and services to help AskA services share 
out-of-seope and overload questions ana access question-answer sets from other se^^'ices. 

2. Partnering with professional associations provides many benefits tu both Ask/X service and 
associations te.g., association members who serve as information specialists provide important 
expertise to the K-12 community while gaining profcssitmal development experience). 
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Case Studies in Contributing 

The following case studies from rcprcscntati\'cs of existing AskA services provide real- 
life stories about managing and operating an AskA scr\’ice. 

• Ask the Space Sciemist: Dr. Sten Odcnwald describes success factors and 
indicators of NASA's Ask the Astronomer and Ask the Space Scientist scr\'ices and 
discusses issues for the future of these and other services. 

• Growin^f Numbers for AskERlC: Coordinator Pauline Lynch shares 
AskERlC's strategics for managing a growth in questions from 151 to S68 per week. 
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Case 

Study 



Hostinjj ''Ask the Spate Scientist'" 

by Dr. Slcn Odcnvvald 



The Astronomy Cafe {lutp:/Av]v]v2.ari.iici/IioiiicV(f(lcn)valcl/cafc.hinii) offers access to 
many documents about research in astronomy as well as the Ask the Astronomer (ATA) 
service {http://i}}\a^e.y^sfc.nciscL^ov/poetry/astro/qaiidcLht}}\l). A similar site. Ask the 
Space Scientist (ATSS) {litlp://hnp:\\'\v\\'2.ari.nci/lu)nie/odcin\'ald/qac/ir/ciancia.himl). 
v\'as developed for the NASA IMAGE satellite program as part of their Education and 
Public Outreach service (residing on the IMAGE/POETRY site: 

litip://i}}ia\^L\i(sfc.}iasa.^oy/p()eiry/), Following 32 months of operation at the Astronomy 
Cafe and 12 months at the IMAGE/POETRY site, over 4500 questions have been logged 
and answ ered at ATA and ATSS at a rale of about 30-50 per week day. Since detailed 
monthly statistics of site activity arc automatically recorded, it is possible to analyze 
some of the parameters of the ATA/ATSS service that scciri to affect its success and to 
anticipate changes in service and site management Tor the future. 

Success Factors and Indicators 

Shortage of Qua} ity Astranoniy-RclatL'd Ash\ ,SV/T/c‘(^v 

CuiTcntly, there is a shortage of astronomy-related AskA seiwices that acti\cly answer 
user questions. Among astronomy-related Web sites, some arc unresponsive with a static 
compliment of questions, others im itc visitors to pose questions, and still others arc 
unreachable due to invalid Web addresses, 01* those that solicit questions from visitors, 
few archive the questions and answers for the benefit of subsequent visitors. 
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There are some notable dilTerences between ATA/ATSS and many of the other on-line 
astronomy services. At ATA/ATSS: 

1 . Both the question and the answer are archived and hyperlinked to several master topic 
indexes 

2. All questions are answered no matter the level of seriousness, and an effort is made to 
respond to questions within a 3-4 day lime period 

3. There are very few large image files that take prohibitively long times to download 
(moreover, the text files are usually no longer than 5000 bytes in length), and 

4. ATA/ATSS is maintained by a single astronomer rather than by a group. This 
streamlines the logistics of responding to questions, and insures that answers are not 
over-specialized. 

Active Advertising of the Site 

It is important to advertise a site .adequately so that a wide enough audience ^s aware of it 
when searching for information on the Internet, The Cafe launched its advertising 
campaign on August 7, 1995, when its address was announced to the Netscape and 
National Center for Supercomputing Application (NCSA’s) "'What’s New” pages and 
with the popular Yahoo (http.V/www. yahoo. cotti) and Infoseek {http:/Av\v\v.i}ifoseek.co?ti) 
indexes. By August 29, the Astronomy Cafe was formally added to these services. On 
September I, the first few questions to ATA were received, and on September 4, NSCA 
selected the Cafe as the Pick of the Week. This gave a favorable review of the site, and 
most importantly, the NCSA review included an embedded link directly to the Cafe. This 
e\’cnt produced a large initial increase in visitors to the site. 

Following this initial recognition, the Cafe received several additional awards and 
review's in the next eight months including: 

• 'I'he “Compulcr Wise” television program earned a review of the site on its 
September 20 program 

• Point Communications awarded the site its “Top 5 9r” of the Intcmet award 

• The “Netsurfer's Digest” (httpi/Avwwjietsnrfconi/nsdl) gave the site very 
enthusiastic reviews as well as another embedded link from their review' page 

• The McKinley Group awarded the Cafe a ”3-star” rating 

• Sky and Telescope {liftp:/Av\v\v.skypiih.rom) added the Cafe to its home page as its 
only link to an external site, and 

• New' Scientist re\ iewed the Cafe at its “Planet Internet” site 
(http:/Av\v\\\ne\\'scientist.c()in/ps/daily/site.html) and awarded it the “Best Site of the 
Day.” 
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Mainiainin^ a Coniforiablc Levt^I ofSen'ice 

In terms of advcilising ATA and ATSS to interested parties on the Internet, this process 
has now reached close to its maximum spread and visibility. However, in paraphrasing a 
familiar adage, "No successful Web site will go unpunished," it is possible to become too 
well known. With literally millions of potential visitors, the penally for success is that the 
ATA/ATSS service can be easily inundated with questions beyond the capacity of the site 
to service in a timely manner. The initial traffic rate in August 1995 was about 1-2 
questions per day. This w^as a very comfortable pace handled by a single astronomier 
working a few hours a week. Once the ATA/ATSS sites became better knowm, a rate 
between 30-50 questions per day became common during the Monday-Friday period; this 
pace requires a more serious load on the astronomer, averaging about 1-2 hours per day. 
When designing such services for the Internet, some serious consideration must, 
therefore, be given to adequately staffing the site against the contingency of its own 

SUCCC.SS. 

She Stafisiics 

Statistics for ATA/ATSS show that these services arc reasonably popular. The statistics 
show a steady increase in traffic; following an initial three month start-up phase, the 
percentage of site visitors who read or post questions has leveled off at about 607c . The 
remaining traffic goes to the other areas of the Cafe and IMAGE/POETRY Web sites. 

0\‘crall, there has been a steady increase in both the number of documents e.xamined by 
site visitors and in the percentage of visitors who ask questions. CuiTenily, one out of 30 
visitors actually asks a question, and the Cafe and IMAGE/POETRY sites attract about 
3600 visitors per month. 
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Site Improvements and The Future 

Increase in Users 

Dcm (1996) declares ihul for the next few years, the capacity and services of the Inlemel 
will continue to increase and improve as new technologies arc brought to bare on the 
problems of improved packet switching times and bandwidth. For at least the next few 
years, the doubling time of new' Internet users is expected to remain near 12-14 months, 
leading to more than 100 million computers connected to the hiternct by the end of the 
decade (Herbst, 1995). This means that the potential audience for any Ask the 
Astronomer acti\ ity is truly astronomical. 

The experience of offering the particular Ask the Astronomer service described in this 
case study indicates that a similar growth trend may be present among the visitors to the 
Cafe and IMAGE/POETRY sites. In terms of education opportunities, it is heanening 
that thi.s kind of service has not yet reached a plateau. By the same token, it is troubling 
that the level of cffoit now required to handle 40-50 questions per day by a single 
astronomer will not suffice to handle an addilional doubling of the site activity without 
seriously affecting the style and content of the pre.scnt resource. 

Advances in Technology 

In addition to the potential increase in u.scrs. there is an inevitable growth of software and 
hardware technology that applies pressure to developers of Web sites. Some sites have 
for scN'cral years incoqvoratcd limited CGI forms-based interactivity which allows visitors 
to enter data into document fields and submit the document for processing at the host's 
machine. Since the inception of the Cafe and IMAGE/POETRY sites, which were written 
in HTML vl. 1, powerful browsers such as Microsoft's Explorer 3.0 and xNetscape’s 
Navigator 3.0 have also come into increasing usage; these browsers can handle HTML 
v2.02 with its extended capacity for multiple fonts, colors and table design. In addition. 
Sun Microsystems' Java language allows small animation programs to be downloaded 
along with a Web page and automatically run on (he visitor's computer as “applets." 
HTML 3.0, which now appears at an increasing number of sites, includes equation 
typesetting. 
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Future Ideas and Clialleiu^es 

It is easy to imagine an ATA/ATSS site utilizing many of these innm’aiions; for example, 
the site could include an animation of the moon orbiting the earth w hilc the text portion 
of the document describes die origin of the lunar phases. Ev en the evolution of the sun 
during the next 6 billion years could be attempted in a series of sequenced static images 
show ing its evolution off the main sequence. 

Currently, the penalty paid for implementing these increasingly elaborate options is the 
increase in download time, requiring that users have faster modems. A purely text-based 
Ask the Astronomer site, on the other hand, may download quickly but lack the visual 
appeal. Most K-12 schools continue to have irlativcly low>cnd technology (c.g., low- 
quality modems below 14,400 baud), and do not have the lime during classroom hours to 
download elaborate graphics, clickable image maps and animations. Other visitors may 
have hiah’Cnd svstems and even T1 data lines. 

Because of the technology options and the discrepancies in user technical capabilities, it 
is important to consider the v isitor you arc trying to serve. Although the cuiTcnt 
ATA/ATSS site w'as intended for use by the K-12 and educational communities, it is very 
clear from the statistics that it is not this community that is using the ATA/ATSS 
resources the most. This means that there is, in fact, an opportunity to further develop the 
Ask the Astronoms resources to match the typicalK' high-end rcstiurccs than many 
visitors in the .COM domain have available to them. 

Developing Res( finx rs 

The ciiiTcni generation of Ask the Astronomer sites are largely text-based. Future ideas 
include local topic search engines, and an increasing array of embedded images, tables, 
line drawings and simple animations. Some pages ma\ even foster visitor interaction by 
prompting for input parameters and displav ing resultant calculations allowing the visitor 
to perform simple numerical cxpcrimcnis. The biggest chalicngc to designers of Ask the 
Astronomer sites will be to keep the resources active and responsive to the di »erse needs 
of the Inlcmct community. Sites that provide a new experience to the visitor each time 
they slop by can become addictive and generate a large cadre of returning v'isilors. These 
sites will also be lime consuming to update on a daily or weekly basis in response to new 
inquiries. It lakes lime to pul lt)gether an elaborate HTML document w ith embedded 
graphics and animation. More importantly, increasing the graphics load of ATA/Ad^SS 
sites increases their download limes iuul this may not he acceptable to many 'iow-ciur 
users, pailiciilarly in the K-12 area. 
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For the foreseeable future, text-rich ATA/ATSS sites will continue to be the mainstream 
offering by astronomers willing to host such sites for the public. Enhanced sites with 
animation and complex graphics will probably remain impractical except, perhaps, as an 
occasional adjunct to the answers of a small number of questions. 



Dr. Sten Odenwald, odenwalci@bolero.f^sfc\nasa.go\\ is the Chief Scientist with 
Raytheon STX Corporation and is the Public Outreach Manager for the NASA IMAGE 
Satellite Program. 
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Managing Growing Numbers of Questions for AskERIC 

by Pauline Lynch 



Case 

Study 



IntroductEon 

AskERIC, the question-answer service of the Educational Resources Information Center 
(ERIC), provides education information to teachers, librarians, counselors, 
administrators, parents, and others. AskERIC began in 1992 and has since seen an 
increase from 151 questions per week to 868 questions per week. The service has 
responded to this overwhelming growth in questions by increasing staff, improving 
processes and tools, standardizing responses to users, and experimenting with publicity 
strategies. 

Increasing Staff 

AskERIC began with a staff of two in 1992 and has expanded gradually to over 50 staff 
members in 1998. A core group of general information specialists are based in the central 
AskERIC office at the ERIC Clearinghouse on Information and Technology. In addition 
to being answered by this group, questions arc answered by subject experts from many of 
the other 15 ERIC Clearinghouses (including Elementary and Early Childhood 
Education: Reading, English, and Communication: and Science, Mathematics, and 
Environmental Education) as well as by experts from outside organizations such as the 
Smithsonian Institution. The decision to distribute questions outside of the central office 
was not based only on the sheer number of questions but also on the wide range of 
question topics, many of which required the expertise of subject specialists. 

improving Q&A Processes and Tools 

Since user questions arc routed to several places, we had to create a system to keep track 
of questions and responses. Early on. we began an archiving system in the form of e-mail 
accounts: each clearinghouse carbon copies its re.sponses to one of the accounts. This 
serves us a way to archive responses according to the responding team as well as provide 
access to responses for quality checks by the AskERIC central staff. 

As wc continue to grow, vve find that this manual system of tracking questions is 
extremely time consuming. Aside from recording which clearinghouse or individual the 
question is assigned to, wc also keep track of the month the question was answered. 

While wc arc cumcnlly able to search the archive accounts for previously written 
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responses by dale, subject line, orC'-mail, we still lack a central location to store all 
question-answer pairs. For this reason, we arc cun'cntly in the process of implementing a 
new automation system that will maintain a full record of all question-answer pairs in one 
location and will help us record electronically the individual or clearinghouse to which a 
question is routed and track whether or not it has been answered. 

Slandardizirig Responses 

In 1994, we developed the A^kERlC Systcfti-Wicle Gi(idc\ a question-answer manual for 
all AskERIC questions. Originally, question-answer policy was unwritten, and we gave 
clearinghouses a lot of freedom to answer questions using their own style. Gradually we 
i‘cali/ed that the wide variety of formats for providing information was confusing users. 
For instance, when parts were missing or formats were changed we started to receive 
questions from users saying they received only a partial answ'cror that they did not 
understand what was sent. The System-Wide Guide still allows for personalization but 
within a uniform framework. For example, the guide includes a set of standard response 
templates for different situations (c.g., nature of question, type of resources used to 
answer question, etc.), but respondents are encouraged to modify any standard statements 
based on the specific situation. 

Fublicitv Strategies 

Publicizing the AskERIC scr\'icc is extremely new for us. Until last year we advertised 
the service by exhibiting and presenting at national and local education-related 
conferences. In 1997, we found that the number of questions paralleled the previous year 
rather than having increased; we began to brainstorm ways to advertise the service. We 
found that announcements on listservs did not draw much traffic, but placing a banner on 
\’arious pages of our Web site nearly tripled our question load. Since our staff was unable 
to keep up wdth this level of demand, the banners — reading ‘'Got an Education Question? 
AskERIC” — were taken down after a couple of days. At the same time, we began 
including AskERIC promotional materials in all orders sent out from the ERIC 
Clearinghouse on Information and Technology (e.g., books, digests, etc.). This seems to 
have continued the trend but at a much more manageable level (207r more questions per 
week). 
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Conclusion 

While a constant increase in questions alTords many challenges and changes, it is a sign 
that our service is valuable to the education community. The cITects of publicity arc very 
difficult to control and may cause sudden increases in questions. Be sure to have a solid 
organizational structure before you look for ways to increase the question load. We 
learned over the past five years that starting out small and gradually increasing our staff 
and question load over a period of time seems to work best, 
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Evaluating 



Like any type of reference service, AskA services should be evaluated regularly in order 
to ensure quality improvement and to justify overall efforts of the service. At the 
conclusion of this module, you will be able to achieve the following goal; 

Goal Plan and implement evaluation of an AskA service and use 

results to improve service. 

Prerequisites Before evaluating an AskA service, an organization should be able to 

1 . Collect information on the general digital reference field and 
existing AskA services (see Module 1: Informing). 

2. Determine the best way to build and maintain digital reference 
service within the organization (see Module 2: Planning). 

3. Plan, produce, implement, and manage training programs for 
service staff and information specialists (see Module 3: 
Training). 

4. Create a prototype and conduct a pilot test of an AskA service 
(see Module 4: Prototyping). 

5. Manage service development and operations and build 
partnerships to gain necessary support and share 
processes (see Module 5: Contributing). 

Objectives At the completion of this module, you will be able to achieve the 

following objectives: 

1. Understand the importance and puq^osc of evaluating an AskA 
service. 

2. Identify areas of the service to be evaluated. 

3. Identify standards by which to judge quality of service. 

4. Identify methods for obtaining information to evaluate an 
AskA service. 

5. State issues involved in planning an evaluation for an AskA 
service and applying results. 
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6.1 Evaluating Ask A Services 



I'hc field of digiud reference inlroduecs now aspects ol' reference service c\'alualion: 
assessing the quality of digital question-answer transactions and reference resources from 
a distance. Digital reference seivices can learn from the research and implementation of 
traditional lihrary reference c\ aluaiion — in the conte.xl of both large-scale and individual 
site studies. 

h'rom 1965 to 1993, over 59 studies w ere eondueied to test the quality of reference 
ser\iees {Saxton, i997).“^ Most of these studies focused on the accuracy of responses 
(Saxton. 1997), while there has been some emphasis on other issues, such as user 
satisfacii(')n. cjuality of refei'cnce intciwiew process, and individual reference lihi'arian 
performance. 

Digital reference evaluation can focus on stmic of the same issues, but they arc translated 
into different terms: quality of ini'ormation specialists' responses, user satisfaction with 
respt^nses and question-answer process, and information specialist satisfaction with 
response process and other scr\ ice policies and procedures. In addition, reference sciw’ice 
in a \ ii'lLia! context implies differences in quality criteria, applications of research 
methods, target audiences, etc. 

This section di.scusscs the purposes for digital reference evaluation and possible areas to 
be e\aluatL'd. Comparisons to traditional librar\' reference cvaliialion issues and 
techniques are included as approi)riate. lhnpha.^is is not on quality eritcria for digital 
relerenec. [)iscussion of quality eharaetcrisiics can be found in Module I: Informing 
(“I'aeels of Quality for K-12 Digital Rcfcrenec Services”). 

Purpose of Digital Keference Evaluation 

The j)rimar\- reason for conducting j’ormal c\ alualitni of Ask A services is \o ensure that 
goals of the scr\ ice and organization are being met most offieicntly and cffeetivcly. It is 
\ei\ easy to get caught up in the dail\ operations of question answering, resource 
building, expert recruiting, etc. and to take for granted that the service is adeciuatcly 
meeting user needs and is being conducted in the most ct'isl- and resiiuree-et ficieiU 
manner. In fact, many sci\iccs do not conduct I'ormal e\alualions but rather rely on other 
channels idclcclors) including user feedback or sponsor requests to guide decision 
making. 



* l oi «i lisl oi lotcroiH’c >ciMvo LW.ilu.itH'n sliuiics iii cln«mi»lo^k':il oi'lIlt h\ ilakMtl sduly trom 19U,S-in93. 
SVC S.lUnM < 1007 I, .7S V7.SS. 
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This module discusses evaluation in a different context from evaluation in the Prototype 
phase (Module 4). The pilot test is conducted solely as ^iX format ive evaluation effort, 
meaning that the primary objective is to improve or revise service. The purpose of the 
Prototyping stage is to test and revise service before it operates according to its intended 
plan. This module discusses boih formative and summative evaluation methods that can 
be conducted on an ongoing basis once the service is in full Contributing mode (see 
Module 5: Contributing). 

In this conte.xt, formative evaluation efforts focus on issues of improvement (c.g., IIow 
can the service more effectively meet user needs? flow can training methods be revised to 
better prepaid information specialists for necessary tasks?), and summative evaluation 
efforts focus on some measurement of quality (e.g., To what extent is the service 
achieving its goals? To what extent is the service making the best use of its resources?). 
Results of summative evaluation efforts are used primarily by decision-makers to make 
“go/no go'’ decisions regarding the service (c.g., renewal of funding). Major questions 
asked may include these: Is the service effective in providing information to users? Is ihc 
service consistent with original goals*.’ Docs the service accommodate user nccus? 

Since me concept of digital reference is so new’, there is not much information available 
in terms of best practice forevaluation of digital reference and AskA services. However, 
AskA services can learn a great .deal from research and literature in library reference 
evaluation. In fact, reasons for evaluating .A.skA services arc similar to reasons for 
c\aluating traditional reference services. Allen (1995, pp. 208-9) presents three basic 
motives for e\'aluating rclcrcnce, which can be summari/ed in the following list: 

1 . lmpro\'c services to u.scrs 

2. Improve processes (clTicienc\ ) 

3. Justify benefits of service to decision-makers (c.g., funders). 

More specific motivations for evaluating traditional reference as well as digital reference 
can include these: 

• Low rate of questions are answered 

• Responses arc incorrect or substandard according to ser\ icc policy 

• Spon.sor demands a proposal or report in order to make a funding decision 

• Users indicate dissatisfaction with rospon.scs or other aspects of service 

• Staff show' signs of burnout. 
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Problems requiring evaluation are not always obvious; services should examine various 
areas of service in order to identify priorities for evaluation. The next section, ''What 
Should Be Evaluated?” discusses possible priority areas. 

What Should Be Evaluated? 



As in traditional library reference, digital reference is made up of many interrelated 
components. These components are outlined in Module 2 as detectors, rules, and 
effectors. After a set of areas has been identified for evaluation, it is necessary for a 
particular service to prioritize the areas and select those with the highest priorities for 
immediate evaluation. Table 6-1 outlines areas of service to be evaluated based on 
possible AskA service detectors, rules, and effectors. Some areas overlap across different 
components and subcomponents. 

Table 6-1: AskA Service Structure and Possible Areas for Evaluation 



fcomponent Agcnt/Subcomponcht Sample Areas 
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'Component 


Agcht/Siibcomponci^t 


Sample Areds I 




Resources — 
Information 
Specialists (IS) 


• User perceptions of IS 

• IS performance (based on required skills as result of 
training program; e.g., accuracy in answering questions, 
inclusion of required respon.se components, etc.) 

• Extent of burnout (f ck of motivation) 

• General attitude towards users 

• Increase in .skills as result of participation in service 




RescHirces — 1 ools 


• Effeclivcncss of software for triage, question answering, 
and archivinii 


FlTectors 


Responses 


• Extent to which rcspon.scs satisfy user questions 

• User satisfaction with amount of information provided in 
respon.se 

• Helpfulness of specific response components 

• Readability 

• Compvehensibiliiy 

• Ability to provide users with follow-through strategics 




Supporting 

Information Resources 


• Readability 

• Ability to meet user needs as identified by incoming 
questions 

• Extent to which user privacy is maintained {c.g., archive, 
ete.) 



As the table implies, evaluation of digital reference service can cover more than the 
actual question-answer process. Attention can also focus on supporting information 
resources (e.g., archive, FAQs), personnel performance, and methods forgathering 
information in order to improve service. 



The items above imply assessment for both effectiveness and efficiency of the service. 
Effectiveness 1: measured by looking at how well the service achieves identified goals 
and objectives (Hemon & McClure, 1987). For example, the achievement of service 
goals and objectives can be reflected by user satisfaction and response accuracy. 
Efficiency is measured by amount of resources involved (or costs) in achieving identified 
objectives (Scriven, 1991). This can include the cost of software and the time taken to 
conduct triage process. 

Quest ion o f Quantity 

Another issue that is often measured by both digital reference and traditional reference 
services and can be related to both effectiveness and efficiency is that of quantity: How 
many questions arc answered by the service? How many users are visiting the service 
Web site? 
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Percentage of Questions Answered 

The percentage of questions answered by a digital reference service relates to the criteria 
of exiensiveness (Hemon 8c McClure, 1987) or “how much of a service is provided" (p. 

1 ). In traditional reference service, this can be measured against the number of people 
served by a particular library. In digital reference service where users arc potentially 
scattered across the globe, extensiveness is measured by the percentage of incoming 
questions answered. Since quantity is easily measured electronically, many services keep 
some ongoing statistics regarding numbers of questions received, answ'crcd, etc. 

However, it is important that the information obtained and recorded be used 
appropriately — in other words, quality should not be assumed based on any such 
measurement of quantity. 

Specific questions that may be of interest to Ask A services include these: 

• How many questions come into the system per day, w eek, month, etc.? 

• How many questions arc routed to (or selected by) experts per day, week, month, 
etc.? 

• How many questions arc answered by information specitilists? 

Such measurements of quantity indicate a service’s ability to respond to users' 
information needs. The Virtual Reference Desk docs not consider number of questions 
answ'crcd in its discussion of “Facets of Quality for K--12 Digital Reference Services" 
because of differences in service capacities and available resources. For this reason, 
measurements of quantity do not necessarily indicate service quality (in addition, quantity 
cannot be used to measure other indications of quality such as response accuracy, 
inclusion of necessary components, etc.). However, if a specific question-answer rate is 
indicated in service goals and objectives, then quantity could be used to measure service 
effectiveness in meeting that specific goal or objective. 

Ffficicncy can be measured in terms of the effect of service proces.ses on the number of 
questions answ'crcd. In some cases, a greater question-answer rate may be possible w'ith 
the use of better processes. For instance, different technology used for question 
distribution may increa.sc the number of questions .sent to information specialists per day 
and thus may increase the number of questions answered; or more appropriate training 
may prepare information specialists to answer more questions at a quicker pace. If current 
processes cause a number of questions to go unanswered, users may turn elsewhere for 
service. In this scenario, unanswered questions become a cost to the service; better 
processes (in terms of training, technology, etc.) could allow' more questions to be 
answered, thus increasing efficiency. 
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Mcusuiing and Antily/.ing Web Site Visitors 



Besides measuring question-answer rate, it is sometimes hclj^t’ul to measure numbers of 
incoming questions and numbers of users visiting the AskA service site to view 
supporting resources (e.g,, FAQs, archive, etc.) and submit questions. Many Web site 
owners use the ‘'hif* method to collect statistics on how many users visit a site. 1'his 
technique of data collection consists of measuring the number of times a particular 
document or image is transmitted, or downloaded, by the Internet Service Provider (ISP) 
to the user’s client software. While mtiny services use this method to measure numbers of 
site visitors, it is relatively unreliable in that it does not account for multiple file transfers 
during the loading of one Web page during a single user’s visit. 

User demographics can also be automatically determined to a degree. Auditing programs 
can measure the number of hits from distinct IP addres.ses. Web site owners typically 
distinguish users by their servers' domain extension (e.g.. .com, .edu, .gov, etc.). Using 
this method, it is also possible to determine the countries users arc connecting from. 
Measuring user demographics in this way docs not provide the most accurate or complete 
information, but can provide some insiglu into the types of accounts to which users have 
access (e.g., university server, K-12 school server from a given U.S. slate, comincrcial 
Internet provider). 

When measuring numbers of visitors, or “hits," users should bew are that they are not 
placing loo much emphasis on the popularity of the site. Instead this information should 
be used to record statistics for service planning. F"or instance, services can identify peak 
question limes during the year (especially appropriate for K-12 services that normally 
have highest numbers during in-schoni periods) in order to anticipate staff and other 
resource needs for a given lime. Or they can identify the Web pages, or locations of 
pages, that are most often \ isilcd in order to decide where to place important information 
about the scr\’icc. 

Ferfo nncui cc Assess ffi enf 

Particular attention should be paid to the job performance of information specialists as it 
IS their responses that become the main product of the service. Standards and criteria for 
job performance are outlined in training objectives for each individual service (see 
Module 3: Training). Results of peiformancc evaluations can lead to revision of training 
programs and materials for new staff or development of new job improvement solutions 
(e.g., job aids, refresher training course, ole.) for current staff. 



1 85 lav 



Evaluating 



Module 6 



Impact on Users 

Since many AskA services include some type of user education as a goal or objective of 
service, it only makes sense to assess the change in knowledge, skill, or attitude on the 
part oF the user as a result of using the service. However, this is extremely difficult to 
measure since many users may use the service only once (not always providing adequate 
opponunity for effective Icaiming), and the possible effects caused by a digital reference 
transaction are often affected by many other factors — user’s previous knowledge, 
knowledge gained from participation in other educational activities, etc. Aside from the 
conceptual difficulties, the actual testing process creates many challenges as well. 

While this Staitcr Kit cannot provide solutions or examples for digital reference impact 
studies, it presents this type of study as a possible area of evaluation. Perhaps AskA 
services can work with educators to guide students in interactions with the service and 
measure some aspect of learning as a result of the interaction (e.g., change in knowledge, 
skill, or attitude). For instance, a library-based service could measure users' development 
of research skills as a result of the service. This may work better for services that 
incoiporatc a fairly large instructional component (e.g., provide guidance in problem- 
solving skills, encourage motivation in subject areas) into their responses than those who 
simply provide factual answers or references. 

6.2 How to Evaluate 

This section discusses standards and methods for evaluating digital reference services. 

Standards for Digital Reference Evaluation 

As previously mentioned, evaluation of AskA services should measure how well the 
service meets its identified goals and objectives. However, service goals and objectives 
don't always provide quantitative standards by which to judge specific aspects of service 
{e.g., user saiisfaction, efficiency of software work-flow package, information specialist 
performance and response quality). Standards for traditional reference services are mostly 
very general, leaving individual libraries to set their own standards for quality or compare 
their situations to those of other libraries (Allen, 1995). 

Digital reference services can look to the Virtual Reference Desk's “Facets of Quality for 
K-12 Digital Reference Services” (see Module 1: Informing) to identify evaluation 
questions and possible factors. Although the 12 identified “facets” are not based on 
quantitative measurements, they provide a set of characteristics and features that indicate 
quality of the question-answer process as wxll as overall service development and 
management. Table 6-2 reviews the facets of quality in the context of a digital reference 
service evaluation. 
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Table 6-2: Facets of Quality and Possible Evaluation Questions and Factors 



Facet of Quality 



User Transaction 



I . Accessible 




?. Sets user expcciatiims 



4. IntcracMvc 



Sample Questions for Evaluation 



• To what extent can all interested users 
access the service through channels 
provided (e.g., c-mail address, Web form)? 

• To what extent can all interested users 
easily navigate service Web resources? 



Are questions answered within a limeframe that 
is useful for the user? 



y\rc users informed as to important service 

policies and qualific 'ions such as: 

• Time frame for returning response? 

• Formal of response (e.g., what it will 
include, where respon.se will appear)',^ 

• Rale of questions answered by service? 

• Fxperi qualifications of service, those 
providing answers (information specialists), 
and resources consulted to find answers? 



• Docs the service provide opportunities for 
users to communicate necessary 
information to information SDCciali,sts and to 
clarify vague user questions? 

0 Are mechanisms in place to capture 
important user information such as age 
or grade level, other inh^rnuition 
sources checked, contact information, 
etc? 

0 Arc mechanisms in place to capture 
user feedback? 



Related Service Factors 



• User technical capabiliiies 

• Ihser special needs 

• User satisfaction with 
Internet resources 



• User awareness o!‘ service 
limitations 

• User satisfaction with 
turnaround time 

• IS performance (i.c., 
training must prepare IS to 
respond within required 
time limit) 

• Effectiveness of systems 
and pr(Kcdures forquestituv 
a AVer flow 



• User awarenc.ss of sc(^pe. 
limitations of service 

• IS performance 
(communicating relevant 
information in responses) 

• Extent to which supporting 
Internet resources coniain 
relevant information 
regarding service 



• Ability to collect u.scr 
information such as: 

0 Characteristics and 
demographics 
0 Technical capabilities 
0 Information needs 
0 Special needs 

• Ability to collect feedback 
on service in terms of 

0 Satisfaction wilh 
response and q/a 
process 

0 Satisfaction wilh 
.service Internet 



Benefits gained and 
acliviiies performed as 
result of using service 
(follow-through) 
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Facet of Quality 


Sample Questions for Evaluation 


Related Service Factors I 


User Transactioi) 




5. instructive 


Do information specialists and supporting 
resources provide instruction in subject 
knowledge and information literacy? 


• I lelpfulncss of specific 
response components 

• Ability of response to 
provide users with follow- 
through strategics 


Service Development and Management 




6. Authoritative 


Do information specialists have the necessary 
knowledge and educational background in the 
service's given subject area or skill in order \o 
qualify as an expert? 


• Ability of policy to establish 
required skills for 
information specialists 

• 1-ixlcnt of screening process 
for accepting information 
specialists 

• Availability of continued 
education or training in 
subject area 


7. 'Frained information 
specialists 


Does the service offer effective ('irientation or 
training processes to prepare information 
specialists to respond to inquiries using clear 
and effective language and following scr\'icc 
response guidelines? 


• IS poilnrmancc on the job 

• Increase in skills as result of 
participation in training 
program and materials 

• User satisfaciicni with 
information .specialists' 
response — content, tone, 
etc. (See Module 3.4 for 
additional information on 
evaluating training 
programs for digital 
referenee.) 


8. Private 


Are all communications between users and 
information specialists held in complete 
privacy? 


Ability of policy to del ine 
measures to restrict public 
access to private user 
information in Internet 
information resources and 
rcsptm.ses 


9. Reviewed 


How often does the service evaluate processes 
and services in terms of information specialists* 
responses, user satisfaction and information 
specialist satisfaction? 


• Implementation of regular 
evaluation schedule 

• Nature of evaluatitm 
questions and methods 

• Data analysis and revision 
decisions 


10. Unbiased 


What measures are taken to restrict prormition of 
cornmercia! products or personal or institutional 
opinions in order to protect quality, credibility, 
and efficiency of service? 


• Ability of policy to restrict 
bias 

• Effectiveness of training 
program to prepare IS to 
avoid bias in responses (as 
measured by IS 
performance) 

• Lack of bias in supporting 
Internet resources 
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Facet Quality 



Suiiiple Questions lor Evaluation 



$ervi/c Development and Managment 



In whai capacities does the service oiTer access 
to siippi^riiny resources and intbrnunioii (c.g,, 
archive, h'AQs, references related information 
sources)? (See Module 5. Coninhuiinii) 



Udalcd Service Factors 



• User satisfaction with 
Internet inforination 
rest>iirccs (type, ft)rnial, 
content, usabllii\') 

• Quality j)f rcMMirces 
0 Readability 

0 Ability it) meet user 
needs as identified by 
incoming qucslioiiN 
0 I'Aiciu to which user 
privacy is maintained 
0 Accessihility (c.g.. 
adequate search 
engines, organisation of 
inlormation) 



2. FAiblieized 


To wh:it extent docs the service inhwir, potential 


• 


User awareness of scope 




users of the value that can be gaiiicti from use of 




and limitations of ser\ ice 




the sci\ ice? (See Module 5. Coninbuiing) 


e 


Quantity (d' incoming user 
tiucsiioiis 






• 


Implementation of regular 
pul)lieiiv plan 




The following methods arc borrowed from lilcraKire on traditional library reference 
evaluation and experiences of existing AskA services. 'Fhey arc presented as suggestions 
and examples. 

"[ hiohirusivc" livalttafion 

Unobtrusive reference service evaluation refers to studies where lest subjects (usually 
reference librarians) arc not aware of the test being eonducicd. Proxies acting as library 
patrons approach the reference desk with questions (to which answers arc prc\’iously 
known) in order to test various aspects of reference service (e.g., response accuracy, 
c|ueslion-ncgolialion skills). 

Backgroim d 

The library cominuniiy has been involved in the research and practice of unobtrusive 
reference service evaluation for about 30 years. In 1%8, Terence Crowley developed an 
unobtrusive evaluation technique that v\ as later revised by ^Fhomas Childers ' (Allen, 



■' See C'rowlev, 'rcrenec and Childers (1971 ). 
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1995). Since then, many unobtrusive reference studies have been conducted"^’ (over 20 of 
which identified a 55% comect response rate to factual reference questions [Allen, 

1995J). 

Advantages of unobtrusive studies in library reference according to Allen (1995) and 
Murfin and Gugclchuk (1987) include these: 

• Since staff is unaware of assCvSsment, they arc able to respond as they would during a 
usual reference transaction. 

• Results can be used to identify areas requiring staff training. 

• Studies can be adapted for use in a single site (library). 

Unobtrusive studies in library reference have been met with some criticism over the 
years. Among the arguments of those opposing such studies include (Allen, 1995; Murfin 
&. Gugelchuk, 1987) these: 

• The fact that test subjects arc not informv^d raises ethical concerns. 

• Questions are mainly factual and fail to represent wide range of typical usr 
questions. 

• Studies mainly concentrate on accuracy of response as the single factor in reference 
transaction. 

Anticipated Uses of Unobtrusive Study in Digital Reference 

Unobtrusive evaluation can be easily adapted to a digital reference context and for an 
individual AskA service. This type of study may even be better suited to unobtrusive 
evaluation than traditional reference. Instead of phy::ically approaching a reference desk 
or making a phone call, the proxy can e-mail a query to a service (or group of services) in 
order to test several aspects of the service such as: 

• Accuracy of response 

• Tone and clarity of response 

• Turnaround time 

• Interactivity (e.g., query negotiation) 

• Setting of expectations for users (e.g., nature of response, turnaround lime, etc.). 

In addition to participating in a digital reference transaction, a researcher can also review 
a sample of previously answered questions in a service's archive — anothe r form of 
unobtrusive evaluation. Unlike most U'aditional reference transactions, many digital 
reference transactions are recorded and archived somewhere — whether or not the archive 
is made available to the public. While this less interactive type of analysis does not allow 



hor iin overview of unobtrusive evaluation in library reference .services, .see Crews (1988). 
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the researcher the experience of an aclual question-answer transaction, it still provi Jes 
adequate information regarding the ntiture of the response including all communications 
that ensued (a* suming all pieces of the transtiction arc kept intact in one message or 
related string of messages). 

In fact, service administrators may use this method on a regular basis or during training to 
assess the performance of infonnalitm specialists. If an information specialist’s main job 
task is to write responses, this sort of assessment can provide valid data on his or her 
performance. 

Although the authors arc currently unaware oj'any formal external unobtrusive digital 
reference studies, they arc aware that some agencies and individuals have tested various 
services for the purposes of their own research and publication. For instance, jouiTialists 
have submitted fictional questions to services in order to gather information for reviews 
of educational Internet resources.' However, like reference librarians who are often 
surprised and disappointed upon finding that researchers have tested them '‘behind their 
backs,” AskA service administrators also feel slighted when reviewers use results of a 
limited number of question-answer transactions to represent the quality of overall sc -vice 
to a public audience. 

User Survey Method 

While unobtrusive evaluation methods can measure many things regarding quaiiiy of 
information specialists’ responses and the .'csponsc prcK'css, they lack the ability to 
accurately assess user satisfaction with the service or inldrmation specialist satisfaction 
with various aspects of the scr\’ice (c.g., training, question-answer process, etc.). I’hc 
method of employing user surveys allows services to gather afl'cclivc data and 
suggestions on specific reference components from large groups of stakeholders and 
audiences. 

In traditional library reference evaluation, user surveys have been distributed \o gather 
feedback on satisfaction with the result of the reference transaction, the reference service 
provided, and the attitude, abilities, and other characteristics of reference staff. 
Limitations of such evaluation methods include the presence of response bias, or the 
inclination of users to react as they think they arc expected to. Because of the possibility 
of response bias and the fact that survey instruments arc so difficult and time consuming 



Hays ( 1997) and “Ask A Librarian...'* (1998) test ihe KidsConncct quest ion -answer service in an 
unobtrusive manner. 
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to LM'calc, it has hccn suggested that libraries use standard instruments " instead of 
designing their own (Allen. \^)^)5), Currently, there arc no standard survey insimnienis 
for evaluating AskA services, but this section can provide suggestions and examples fur 
creating original instruments. 

Designing Questionnaires lor Reference livaliiation 

Morgan ( D/05, pp. 142-147) offers the follow ing rccmrimcndations for ci-eatmg sliiacvs 
to evaluate reference service performance assessment: 

• Keep it short 

• Include all relevant questions 

• Include adequate white space between questions 

• Include questions of interest to respondents [o encourage further participation 

• Keep instructions consistent (e.g., use one type of “check” method for dosed 
tjiiestions, such as circling, clieck mark, etc.) 

• I -sc simple language, free of jai'gon and acronv’ms 

• State questions dearly in order to coniiminicale their exact meaning (e.g., avoid 
dt)Lible neuatives, ambi^uous wordiim, etc.). 



Survey Ifse in Digital Referenee 

Distributing and anal}7.ing surveys from users of digital leference services is rdati\d\ 

easy and has been useful in helping AskA services identify areas for rev ision. ALbaiUagi's 

m conducting digital reference tiscr surveys include these: 

• Surveys can be distributed directly to e-mail accounts of users who have previously 
asked questions of the service, making the survey directly accessible. 

• Surveys can also be created as Web forms, also making tlie survey easily accessible 
and allowing the creation of scripts to automatically tabulate responses. (Note: 
Services should restrict access to i\ Web-based survey in order to I'ilter out those who 
have previously pailieipalcd in a question-answer transaction.) 

• Users cun I'cturn surveys simply by replying to an e-mail message or automaiicallv 
submitting it from a Web form, possibly increasing the icturn rate over paper siir\c\s 
that have to be mailed or physically delivered. 

• Surveys can be sent to users in such a manner that a previous question-answer 
inleraclion is included in the message — this can help refresh users’ memories 



One standard survey Itirm for library rcl'ci cnee assvssmcnl is the \Vise()iisin-C)hit> form. See liunr.e 
( 1985) and Miirf in and Cmgelehuk ( 1987). ^ ^ 
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regarding Iheir use of the service and help ihe scr\ ice analy/c user res[H)nscs in ihc 
ccMilexi of users’ acUial experiences. 

• Conicni of surveys can focus on several specific aspccis of service (c.g., componcnls 
of message, informalion specialists' lone, Uirnaround time, eic.) iis well as user 
demographic and technical informalion. This provides services with a largeicd 
overview of user satisfaction and user rcquiremenls of service. 

Common issues largeicd in surveys for digital retcrcnce include these: 

• User informalion 

0 User demographics (c.g., grade, profession) including capacily in vv hich 
qiieslion was asked (c.g., a lihraw media specialist may ask a question for 
professional development or n'ay send a question on behalf of a third-grade 
siudcnl) 

0 User technical capabilities (c.g., ability lo conlaci serv’ice. access response, 
etc.) 

C> Abililv' lev Li.se informalion or lo access resources referenced in respcuise 

0 Number of times user has submitted quesliims to ihe service (for idenhficalion 
of repeal users). 

• SalisfaclicMi with response content and process 

0 lixlenl lo w hich response answered user queslitm (e.g., relevance or 
effectiveness of information prov ided) 

0 Satisfaction with amount of informalion included in the message 

0 Helpfulness of standard response components (e.g.. rc.sources referred, 
inclusion of user question, etc.) 

0 Readability and comprehensibility of response 

0 Satisfaction with turnaround lime of rcspon.se 

0 Consistency of response in terms of user exj-)eclalions. 

5 I'ollovv-thmugh 

0 C'iipacitv m which response will he u.scd (c.g., schoed assignment, perscmal 
interest, lesson planning). 
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• Publicity 

0 Information source that originally informed user of service (e.g., Web site, 
brochure, conference presentation, etc.) 

0 Other individuals with whom user communicated regarding question creation 
or response contents (c,g.. teacher, colleagues, etc.), 

• General feedback 

0 Willingness to use service again 

0 Benefits gained by user as a result of service (e.g., save time, money, etc.) 

0 Open suggestions, comments, etc. 

Some services may also choose to include questions relating to accessibility and 
usefulness of Internet resources such as archive, FAQs, links to related sites, etc. 
However, some services wish to focus questions on the question-answer process 
separately from related resources. 

An example of a questionnaire used to evaluate an Ask A service is found in Figure 6-1: 
AskERlC Q & A Survey. 

Besides questionnaires, other survey activities include discussion groups and interviews. 
These methods can be used to capture the same type of information as a questionnaire, 
hut they add an interactive clement. During these interactive sessions, questions and 
responses can be clarified, respondents can communicate nonverbal reactions, and 
conversation can lead to more in-depth issues (Morgan, 199,S). In a traditional library 
situation, discussion groups and interviews arc usually facc-lo-facc meetings of one or 
more users and facilitators or interviewers. In digital reference settings, the conversations 
can occur virtually. For instance, a service can set up an expert panel of representative 
users to launch discussions about responses, Web resources, processes, and other issues 
identified for evaluation. This .should be done with a controlled group of users who 
express interest in participating in such a survey. 
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Figure 6-i shows a poriion of the AskERIC Q&A Survey conducted in 1998. 



Figure 6-1 AskERIC Q&A Survey 



AskERIC Q&A Survey 

Thank you for recently using the AskERIC Question Answering Service. In order to help us 
better serve our patrons we are conducting a brief survey. Please take a few minutes to answer the 
following questions. Your help would be greatly appreciated. 

AskERIC respects the privacy of its users. Participation in the survey is voluntary, and all 
information will be kept anonymous and confidential. Your name will not be used in any way, 
and no one will know about your specific answers except the study researchers. 

Thank you for your time and cooperation. 

The AskERIC Staff 

1 . How did you send a message to AskERIC? 

A. e-mail 

B. Web form 

2. What is your profession? 

A. librarian 

B. K-12 teacher 

C. pre-K teacher/caregiver 

D. administrator 

E. student 

F. faculty 

G. parent 

H. government official 

I. other 

3. In what capacity did you request information from AskERIC? 

A. librarian 

B. K"12 teacher 

C. pre-K teacher/caregiver 

0. administrator 

E. student 

F. faculty 

G. parent 

H. government official 

1, other 



( I 
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4. Ha\c you used AskERIC' more iliaii once? 

A. yes 
H. no 

4. 1 If VOS, how many limes have you used the serviee in ihe pasl year? 
' A. 1-3 ' 

B. 4-6 

C. 7-10 

D. more than 10 

5. Was ihe infoniiaiion provided relevaiu lo your quesiion'? 

A. yes 

B. no 

0, (Did die inlbrmalion prox ided lead you lo an atiswer lo your queslion? 

A. yes 

B. no 



7. How are you using the in formal ion gained from ihe ser\iee‘? 

A. research 

B. classroom implemeniaiion 
r. personal 

1). career preparalion 

E. olher 

8. f las ihe serviee saved you lime? 

A. ves 

B. no 

8.1 mone\? 

A. yes 

B. IK) 

9. Please rale ihe helpfulness of ihe differenl eomponenls of an AskERlC' response. ( I=LE.AST 
HELPFUL, = MOST HELPFUL. NA =NOT AI^PLICABLE). Please noie lhai not all 
eomponenls are presenl in every response. 

ERICciiaiions 12 3 4 5 NA 

Internet sites 1 2 3 4 5 .NA 

Listservs 12 3 4 5 NA 

Organizaiions ! 2 3 4 5 NA 

Ollier Resources 12 3 4 .5 NA 

10. Was ihe in formal ion easy lo read? 

.A. yes 
B. no 

10.1. Wh.y/Why not? 
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Other Methods for Measuring Staff Performance 

Aside from unobtrusive testing and user survey dislnbulion, there arc other possible 
methods for directly c\'aluating information specialists' performance in creating digital 
reference responses. (Also sec Module 3: Training, for more suggestions on evaluating IS 
performance during and after training). Possible methods include the following (scr\ ices 
can combine the methods if desired): 

• Peer assessment — AskA services can establish systems whereby senior informaiion 
specialists (those who have demonstrated quality performance) assess the responses 
of newer information spccia.lists and provide feedback on their performance. Some 
services implement this method by splitting their information specialist (cxpcrl) base 
into distinct groups (based on expertise area or some olhci’ grouping). Leaders or 
moderators of each group regularly assess group members' responses according to a 
set of criteria (c.g.. inclusion of required response components, friendly tone, etc.) 
and then provide each informaiion specialist with feedback to help them improve 
future responses. 

• (irading or Tracking — (nxiding or tracking schemes can be implemented in order to 
assign values to informaiion specialists' responses. This method allows services to 
judge individual informaiion specialists in a consistent manner. Informaiion 
specialists can be assessed over time to track improvement and make decisions 
regarding advancemcni or discontinuation with service. (For an example of a grading 
scheme for AskA service rcspon.sc.s, sec "The MadSci Moderators' Manual" 
laip:/Av\v\\\})uidsci.ori>/ask_cxperi/})\odcniiors.hw\l) 

• Self-assessment — Some services may choose to implement a program where 
information .specialists evaluate their own performance according to predefined 
standards. Some form of recording should be in place lo help information specialists 
track their performance and report results lo sci'vice administrators. Recording can be 
facilitated with insirumcnls such as checklists to be filled out after each response. 

Regardless of the methods used for performance evaluation, it is important lo inform 
information specialists (or other staff) of their level of performance and suggest ways for 
them to improve. (For suggestions on providing feedback, sec Module 3.4). Common 
issues derived from pciTorinance evaluations (c.g., all informaiion specialists fail to 
include a required response component) indicate that there may be a need to revise 
training programs or reinforcement resources and activities for current staff. 
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6.3 The AskA Service Evaluation Process 

Hvaluaiion of .AskA scr\ iccs .should be included as pail of the o\ crall scr\icc plan and to 
be most c[Tccti\’C should occur on an ongoing basis. CuiTcir’y, most services do not 
conduct ongoing formal evaluation acti\'itics because of constraints on capacity and time: 
It is suggested that services staii slowly by focusing on one or two areas and then build 
on once comfortable methods and strategics are in place. Of course those services that 
have pressure from funding organizations do not always have the luxury of choosing 
evaluation areas and schedules. In any case, it is important for services to always keep an 
eye out for potential evaluation areas (sec Tables 6-1 and 6-2 for sample areas) so that 
they can be properly examined at the appropriate time. 

Planning Issues 

Once a service has identified target issues and questions for evaluation, it needs to plan 
the evaluation in order to ensure that the goals are achic\ ed. There are many issues to 
consider when planning an evaluation in any context. " ^khe steps below are described in 
the context of digital reference evaluation: 

1. Creation of evaluation goals. I'hcre should be a defined purpose for conducting the 
cNaluation (aside from the charge by a sponsoring organization). The goal should reflect 
larger service goals and objectiv'cs and should indicate the target evaluation areas. Below 
are examples of evaluation goals for AskA scr\’iccs: 

• Measure user satisfaction regarding standard components and language used in 
responses. 

• Assess performance of information specialists in completing the response process 
according to service policies and procedures. 

• Evaluate the usefulness of work-flow .software in terms of routing questions to 
information specialists, assisting information specialists during the response process, 
and archiving question-answer sets. 

2. Formation of evaluation team. The team approach to planning and implementing 
evaluation of digital reference services is beneficial to the overall effoii. As is true of 
traditional reference service evaluation, digital reference evaluation cffoils arc frequently 
loo large for one staff member to accomplisli individually, and it is helpful to gain 
perspectives of different people both internal and external to the service. By inviting 



' Thi^ section IS adapted trom Smith and t'ostcllo ( See Mcrntm and McClure ( 1987) for discus.sion 
of steps involved in establishing formal evaluation syslenis for reference servico.s (pp. 1 12-128). 
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input from service staff — especially those whose w'ork is being evaluated — a sense of 
trust of the evaluation effort can be established. When forming the evaluation team, 
services may consider outside researchers who can provide an objective approach. This 
can include a supporting organization’s evaluation department, graduate students at a 
sponsoring university, etc. 

3. Identify major questions and sub-questions. Besides establishing goals to guide the 
evaluation, services should identify general and specific questions that they expect the 
evaluation to answer. For instance, services may wish to know if users are generally 
satisfied with responses to their questions. More specifically, a service may want to 
examine to what extent users are satisfied with specific components of their responses, 
such as the list of references included, instructions for accessing resources, and the brief 
factual answ'er. 

4. Select procedures or methods to collect information. For each question and sub- 
question identified, services should establish a plan for collecting information to address 
it. For instance, in order to find out if information specialists are using clear language in 
their responses, the evaluation team may propose to review a sample of archived 
responses. 

5. Establish data collection schedule. It helps to establish a time period in which 
evaluation activities will take place. This assists the service in completing activities in a 
realistic and useful timeframe so that results can be used to make necessary decisions. It 
also helps to balance multiple activities that are sometimes required of evaluation efforts. 

6. Identify types and sample of respondents. If planned procedures require the 
participation of individuals to provide information (e.g., questionnaires, etc.), the 
evaluation team should identify who and how many should be asked to provide 
information. For instance, should a user satisfaction questionnaire be distributed to all 
users who submit questions and receive responses in a given period of time? Should it be 
distributed to only a sample of users who receive responses (e.g., every third user)? How 
should users be chosen to participate in an expert panel discussion group? 

7. Create plan for analyzing and interpreting information. Once information is 
received through the methods pursued, it must be analyzed and interpreted in such a way 
tliat evaluation questions can be effectively answered. Information received should be 
compared against evaluative criteria. For instance, the criteria for efficiency of work-flow 
software could include that 907c of incoming questions must be routed to information 
specialists on the day they are received, 1007r of information specialists’ responses must 
be automatically saved in the archive, etc. 
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8. Create evaluation managenient plan. Besides planning a basic schedule of 
activities, larger evaluative efforts may require plans that outline the budget and 
personnel needed to conduct the evaluation. 

9. Outline reporting methods. Depending on the purpose and goals and evaluation, 
reporting methods can include brief internal reports for the service staff reference or 
formal research reports fora sponsoring organization. Regardless of the specific format, 
evaluation reports should be written clearly, in the context of the intended evaluation 
audience, and should encourage some type of action in response to the evaluation results 
(Hernon & McClure, 1987). It is helpful to establish a plan for reporting before the actual 
evaluation occurs in order to ensure that reporting activities arc conducted appropriately 
and in a timely manner. 

Using Results 

Once the evaluation has been conducted and information has been analyzed and reported, 
results can be used to improve scr\'icc. E\'alualion results do not always provide clcar-cut 
solutions for problems identified but can be helpful in stalling initial discussion regarding 
ways to improve the service. This discussion should involve staff affected, and priorities 
should be set for developing and implementing solutions. 

E.xampics of solutions based on c\ aluation results arc presented below’: 

• Evaluation indicates that users have difficulty understanding some of the content that 
is regularly included in responses (c.g., codes referring to service resources). This 
could lead to several solutions: eliminating codes, pro\ iding definitions for codes, 
spelling out codes, etc. The service may decide to test various solutions and then 
survey users again to see which are most helpful. 

• Evaluation finds that many information specialists refer users to Web sites that arc of 
poor quality and arc inappropriate for the user audience (c.g., K-1 2 students). 
Solutions may include a new' component of staff training introducing information 
specialists to criteria for assessing Web sites suitable for inclusion in responses. In 
addition, such criteria can be included in a job aid for current information specialists. 

• Evaluation of user qucsticMis finds that 459r of incoming quciics are out of scope for 
the service. Solutions can include revising the service Web pages and promotional 
materials to more accurately reflect service scope (c.g., including examples of 
appropriate questions) or instituting user education programs (c.g.. prc.scnlations to 
educators, students, and parents) to communicate important information regarding the 
service and appropriate use. 
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Module 6 Summary 

This section provided suggcMions and examples htr possible areas and methods of AskA service 

evaluation. 

• Hvaluaiion of digital reference service is a new area iIkii can learn fmm research in e\ aluauon of 
traditional library reference service. 

• The primary reason for conducting formal evaluatuai of AskA services is to ensure that goals of the 
.service and organization are being met most efficiently and effectively, {-valuations of digital relerence 
sei vices can be both formative (for improvement purposes) and summative (for dccisicai making) in 
nature. 

• In deciding what to evaluate, .ser\iecs should examine curreni detectors, rules, and effectors. Sample 
areas include quantity of questions and users; staff performance; and service impact on users. 

• Standards for digital reference evaluation aic based on the Virtual Rel'ercnce Desk’s ‘'l-aec(s of Quality 
fiM- K-12 Digital Reference Services.” 

• Methods for digital releienee evaluation can include uivohtrusive evaluatitm and u.sor survey. 

.Additional methods are used specifically for staff performance as.scssnient: peer assessment, grading im' 
tracking, and self-ussessmem. 

• 'fhe AskA service c\aliiaii(m process consists cd an e.xiensive but important planning phase including 
the creation of goals, formation of evaluauon team, identifiealit)ii of questions, selection of methods, 
creation of data collection schedule, identification oi respondents, and creation of plans \'or analysis 
and interpretation of information, management of evaluation, and reporting of results. 

• hT aluation results can he used to start discussion of identified problems and possible sidutions. 
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Module Profile 



This module provides a conclusion to the AskA Starter Kit. At the end of this module, you 
will be able to achieve the following goal: 

Goa! Place the Starter Kit into a larger framework that incorporates 

an action-based aeenda and a vision for the future of digital 

w C? 

reference. 



Prerequisites Before reading this conclusion, an organization should have completed 
the six modules representing the steps involved in building and 
maintaining a digital reference service: 

1. Collect information on the general digital reference field and 
existing AskA services (Module 1: Informing). 

2. Determine the best way to build and maintain digital reference 
service within the organization (Module 2: Planning). 

3. Plan, produce, implement, and manage training programs for 
service staff and information specialists (Module 3: Training). 

4. Create a prototype and conduct a pilot test of an AskA service 
(Module 4: Prototyping). 

5. Manage service development and opcnilions and build 
partnerships to gain necessary support and share processes (sec 
Module 5: Contributing). 

6. Plan and implement evaluation of an Ask^X service and use 
results to improve service (Module 6: Evaluating). 

Objectives At the completion of this module, you will be able to achieve the 
following objectives: 

1 . Understand the importance of the human intermediary in 
digital reference service (providing contextual information in 
response to user needs). 

2. Understand the role of tcctinology in connecting users with 
information. 

3. State the main challenges for digital reference service in the 
future. 

4. Apply suggestions and examples from this Starter Kit to the 
development of a new AskA service. 
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Summary and Challenj^es (or Ihe Future 

The implicit promise of ilic inlormalioii age is eoniexl, 'Fhose seeking inrormaiion are not 
looking for VVen pages, hooks, or ariieles: rather, they are looking for answers — solutions 
to a problem grounded ii. dieir siuiation. What is a petirl ol' wisdom one day can he a 
useless jMeec ofti’iN'ia on another. Discerning user needs and dealing context are 
inherently human activities, because they in\'ol\'c the process oreommunieatuMi — not the 
mechanical proce.sscs of archiving and delivery, 'rhcrcl'ore the needed intermediary oT the 
inrormaiion age is not a Web page but rather a human being. 

'I'he goal ol’ any IcchnologS’ is to become iin isible. deehnology is a means to an end. If 
wc arc to fuinil the promise of the inlormalioii age and connect those with pi'ohlems vriih 
those with answers, we must make the technology that wc use transparent. ITmaii clients, 
Weh sites, and modems must appear to vanish as users connect to experts. The paradox 
here is that only by impj'ov’iiig technology, and the processes that utili/.c these 
technologies, can wc make the tools, products, etc, disappear. 

'The great challenge before digital reference sciA’iees (from those who research them to 
those who use them to liiose wiio pro\ ide them) is to aehie\e a seamless integration ol' 
resources, people, and leelinology. 'I'his may well he Plato's unattainable ideal, but it is a 
goal to strive for. rcriainls' digital referenee is in its infancy and has much [o learn as it 
grows. Challenges remain ahead: 

• How do we incorporale real-lime conimiinicalions into our .services? ! Iieie is 
little doubt this vSlarlcr Kit assumes a heavy amount of e-mail and asynchronous 
service. How can we build models w'ith real-time dials and video conferencing? 

• How do we build an economy of reference? 'Phis is more than determining ‘‘per 
question" price. It relates to how we \'aluc experts, manage and allocate resources, 
create barter arrangements, constrain transactions, build policy for the fair use of 
services, and, yes, cost services. 

• What .standardizalioii is needed? What arc the leeliiiieal and jiolicy standards that 
we need to pul in place to create seamless navigation from reference service to 
reference service, and reference service to resource? 

• Is fragmcniation of digital referenee inevitable? Does digital reference need to he 
different for libraries, education, and business? Tliis Starter Kit was written on the 
assumption that the challenges facing all of these sectors arc common and can best be 
explored and .solved together. 
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hi spite ol' these and many other c|iieslions, the outlook Tor dieilal releivnee scr\ lees is 
hrighl. As large organi/aiions ino\'e to the Weh to eonduei iheu* da\ -iO'day (operations, 
traditional areas of customer service, commerce, and ijiialily manaueniL*nl will he 
challenged to adapt lo the Inlcrnel en*» ironmeni. With the entry of large jiri\ale and 
public insliliiiions into the realms of digital reference (he they called lielp desks, eusiomei 
service, referenee, oi’ Ask A) comes increased investment, understanding, and itools. Tiie 
(lueslion I’or the readers ol' this doeumeni is: Do you want to be the early explorers o( 
these domains, or wait for the proldenis lo be solved? ll'you wiiil, are you confident that 
corporations, go\'cmmeni, or other sectors will considei' your needs and \iews as ilie\ 
dejiloy their systems? 

I^y working through lliis Starter Kil you now ha\'o the tools create AskA sci \ ices. 
Ilopefully you ha\'e a iiicture of the needs of your organi/alioii. I\)leiilia!l> \ou are 
returning to this text with a scr\'iee eslahlished. What you must I'cah/e is dial even ihough 
the ink IS dry and the cover bouiuk this book is still being w nlleri. Your exjvrieiiee. sour 
trials, and your resources will shape its evolution. 

Whore to (Jo From Hero 

So, where to go from here? The best answer is sim|il\ to start. Start learning more b\ 
visiting the services discussed in this kit. Start planning by surveying the needs of unw 
organization. Start training in order to gel your experts online and readw Start 
prototyping by downloading sample software or ev’cii an e-mail elieiil, Start coninbutiiiL* 
lo the comiiiuiiily through your ser\ ice--- tell us what vnii're doing, share sour expeilise. 
Start evaluating by asking the hardest cjucslioiis ol' all for those who have an Ask.A 
service — is it being done w'cll, and does it make a differenee? 



Sliiarl Siuioii, a professor at Syracuse University's School of liiloriiialion Studies, once 
said, “'I'he future does not come shmik-wrappod." Projects sc.ch as the \'irlual Kclereiice 
Desk and Ask Dr. Math and companies such as Remedy Corporation are creating 
software for AskA services, hut they have just begun. 'I hc initial jdiia.e of software 
development w'ill be sample code, lest bed iirojccts. and source code that can be adapted. 
IdTorls are underway lo create standards for the iiilerehaiigo of questions and referenee 
interview's. All of these efforts arc in their infancies. I* ven tlie long eslahlisiicd hclji desk 
soluvarc requires extensive eiislomi/.ation (and eonsidcrahle cost) to create a workable 
system. 

Now is the lime lo gel involwd. Now is the time lo pul \our voice and the lU'cd.^ ol \our 
users into a larger dialogue ahuui how lo conduct digital refeicncc. If you're a hhrarv . 
lend your kiiowdcdgc of the reference process. If you’re a husmoss. share your 
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understanding of customer service. If you're an educational institution, share your 
experiences in pedagogy and curriculum. We stand ai the dawn of a communications 
revolution, a revolution that will place context abo\ ; data, and people above Web pages. 
You are invited to be a part of this new era. 



Module Summary 

This section provided a summary of the Starter Kit and a frameNvoi k for digital reference now and in the 

I'll lure. 

• Human intermediaries are crucial in digital reference service in order to determine user needs and 
communicate information in context. 

• Technology for digital reference must be improved in order to appear seamless to the user. 

• The greatest challenge for digital reference services is to achieve seamless integration of resources, 
people, and technology. Other challenges include incorporation of real-time communications, creation 
of a reference economy, development of standards for technology and policies of digital reference, and 
exploration of common issues across various digital reference contexts. 

• Organizations interested in building AskA services can begin immediatcl) using the suggestions and 
examples from the six modules of the “Starter Kit.” Suggested activities include visiting existing AskA 
services, surveying needs of organization, developing a training program and training experts, 
downloading sample strtAvare. sharing expertise w-ith the digital reference community, and p'anning 
Tor service evaluation. 
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About The Virtual Reference Desk 

The Viriual Reference Desk (VRD) is creating the roundations for a national cooperative digital 
reference service. The project is sponsored by the National Library of Education (NLE)and the 
ERIC Clearinghouse on Information & Technology, with support from the Office of wScience and 
Technology Policy. The Virtual Reference Desk seeks to identify and provide the resources 
necessary to link all K-I2 community members (e.g., siudciils. educators, parents, etc.) to 
necessary expertise in order to satisfy information needs. 

The goals of the project include research into current ways in which K- 12 community members 
receive answers to questions on the Internet and development of a national collaborative network 
of Internet-based question-answering services. 

What is Digital Reference?. . . 

Digital reference services, also called “Ask-An-Experi" {or *V\skA") services, arc Internet-based 
question and answer services that connect users with indi\'iduals who possess specialized subject 
or skill expertise. As opposed to sialic Web pages, digital reference services use the Internet to 
place people in contact with people who can answer specific questions and instruct users on 
developing certain skills. 

Resources and Services include. . . 

• Database of question/answer services — The AskA+ Locator contains a collection of 
over 70 quality AskA services for the K-12 community. Services can be searched or 
browsed (///r/;;//uanr. yrd.or^/locator/imiex. html). 

• Virtual Reference Desk Web site {wwwArd,org ) — This central access point for digital 
reference provides information on exemplary AskA services and offers resources to help 
organizations build and maintain new AskA services, including training materials and 
quality criteria. 

• Starter Kit — This instructional resource guides organizations in the development of new 
AskA services in their areas of expertise by providing '‘how-to” advice and methods 
based on experiences of exemplary services and' in-depth research. 

• Dig_Ref Listserv — A discussion group open to those interested in all conlc.xls of digital 
reference service. Cun-cni subscribers include those working in libraries, education, 
business, government and other sectors, both nationally and internationally. 



For more information on' 
the Virtual Reference Desk, contact 

vrd@vrd.org or (800) 464-9107. 





Glossary 



The following icrnis arc defined in Ihc conlext of digital reference scrx'ice as outlined in 
this Starter Kit. 



Archive: A stored collection of qucsiion-and-answer sets. May be publicly accessible 
online or retained for a service’s private use. 

Ask/V Consorlium: A group of AskA scr\ ices in a cooperative network that seeks to 
provide resources (computing infrastructure, funding, and knowledge) to its members and 
guides the overall Viniial Reference Desk project. 

AskA+ Locator: Searchable database oi‘ over 70 AskA services that serve the K-I2 
community and meet some minimal quality criteria as idciiiificd by an expert panel. 
Detailed descriptions arc ax ailablc for each service in the Locator for assistance in 
selecting the most appropriate sei'vice for a given situation. 

Iittp://\y)v\\\\ rd. a r^/h )ca to r/i f tdc.w hrifil 

.AskA services: Internet-based qiicstion-aiul-answcr services that connect users with 
indix'iduals who possess specialized subject or skill expertise. Also known as digital 
reference services, provide human expertise through questitm-answer services on the 
Internet. So called for serx ices such as Ask-A-Scientisl or Ask-A-\'olcanologist, these 
services take questions through e-mail and the World Wide Web. 

Blueprint: Graphical reprcsenlaiion of the conceptual framework and methodology of an 
AskA service. 

Contributing; The ('ingoing process by which an .AskA service offers informali(Mi 
scrx iccs to the user community (c.g.. frcqucniiy K- 12 educators, students, and parents) 
and shares expertise and experience with its supporting organization and digital reference 
community. 

Detectors: Mechanism lor acquiring information about an organization’s environment. 
Information gained is used to ensure that an AskA service meets stakeholder and user 
needs and uses the most efficient moans to accomplish goals. 

Digital rerercnce: Held of information science bK'using on the creation and operation of 
Internet information systems to satisfy user needs. Although many concepts and 
pimccsscs have nxils in traditional library reference, digital reference applies to liuernct- 
based t(ucstion and answer services in many contexts including education and business. 

Digital rclcrence .services: Internet-based question-answer and refcira! services tliat 
reach a wide audience on a wide range of topics. Services can be operated by libraries, 
professional associations, educational institutions, and other types of organizations that 
can connect users with individuals possessing specialized subject or skill cxpcrlisc. Also 
referred to as AskA .services. 
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liffectors: Sets of services offered to users or others on the Inlcmel in order to 
reference needs. 

ERIC (Educational Resources Information Center): National education information 
s\'stcm sponsored by the Office of Educational Research and Improvement in the U. S. 
Department of Education and the National Library of Education. Established in 1966, 
ERIC produces the largest database of education information in the world. 

Evaluation: Systematic process by which the value and efficiency of a product or ' 
process is determined according to set criteria. 

Expert panel: Group of repre.scniativcs from A\skA services and other related 
organizations who identified the quality criteria for K-12 digital reference services and 
selected sc\'cn otcmplary services that fit the criteria. 

Experts: Staff or volunteers possessing special skills and knowledge derived from 
training, education, and e.xpcrience. Also rcfciTcd to as information specialists when 
providing expert information to digital reference service users. 

Extensiveness: Percentage of incoming questions answered. 

FAQ (Frequently Asked Questions): Document containing commonly asked questions 
and their answers. Often provided as part of a digital reference service's Web site. 

Formative evaluation: Method of evaluation that focuses on issues of improvement. 
(Example: How can services more effectively meet user needs?) 

GEM (Gateway to Educational Materials): A metadata project developing 
vocabularies and standards to allow educators to find lesson plans and other educational 
materials quickly and easily on the Inlcrncl. GEM is sponsored by the U.S. Department 
of Education’s National Library of Education and is a special project of the ERIC 
Clearinghouse on Information & Technology at Syracuse University, 
httj)://\v\v\\ \ th cyclic way, o /*g 

Goals: Positioning statements defining what an AskA service expects to achieve. 

(trading or tracking: Evaluative method where alphanumeric values are assigned to 
information specialists' responses in order to rale performance. 

llollancPs Performance System: A means to describe any organization by detectors (the 
means the organization has to gain information about the world), ruLs (the processes an 
organization uses to determine a course of action based on information from detectors) 
and effectors (mechanisms an organization has in place to affect the environment or its 
place in that environment). This scheme of dctcclors-rulcs-cffcclors was developed by 
John Holland of the University of Michigan. 
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Information specialists: Human intermediaries that direct a course of action for 
questions received by AskA services. May also be subject experts to whom questions are 
submitted. 

Instructional design: The systematic process of planning events to facilitate learning. 
Can be applied to planning of training programs and materials for digital reference 
service staff. 

Internet Kngineering Task Force: 'The Internet Engineering Task Force (IETF) is a 
large open international community of network designers, operators, vendors, and 
researchers concerned with the evolution of the Internet architecture and the smooth 
operation of the Inteinet. It is open to any interested individual." 

Iiltp://\v\v u ietf. o rg/ove iew.li tml 

Internet Public Library (IPL); A project of the University of Michigan School of 
Information that provides Inteinet users with access to resources as well as digital 
reference service. hitp://\v\v\\\lpLorg 

Job aid: Supplemental resource intended to help the trainee come up to speed during 
training and to provide support for performance on the job. 

Knowledge base: A distributed index of question-and-answer archives across all AskA 
services in the AskA Consoaium. A planned resource of the Virtual Reference Desk 
Project. 

Key detectors: Most important detectors in forming and maintaining decisions. 

Metadata; Information about data itself. Metadata is used to understand, manage, and 
facilitate the sharing of and searching for materials on the World Wide Web. 

Meta-description: Information about the core work-flow processes and functions in 
digital reference service. 

Meta-triage: Function that is the backbone infrastructure that connects AskA services 
and end users together. It acts as the central switchboard for questions and answers 
traversing the Virtual Reference Desk’s network of users and knowledge resources. 

NLE (National Library of Education): Largest federally funded U.S. library that is 
entirely devoted to education. hap://\vww. ed.gov 

Objectives: Detailed statements describing how an AskA service intends to accomplish 
its mission. Also refers to key skills and knowledge AskA service staff is expecteo lO 
achieve upon completion of training program. 

Peer assessment: The assessment of AskA service staff performance by more 
experienced staff members according to specific criteria and with the goal of providing 
constructive feedback. 
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Pilot test: Preliminary process oi‘ ‘‘trial run" condueicd as a basis for cuilualion in order 
It) improve services. Can be eondueled to e\'aluatc overaM service or components of 
service such as training program. 

Prototypiiij»: I'lie de\elopment of an initial functional, though incomidcle, type or design 
of a product to be refined in sueeessi\’e tests. Digital reference services often engage in a 
prototyping phase in order to test service in a controlled environment before it is offered 
on a large scale. 

Rules: Rules re]')resent the aetit)iis ol'an AskA serx iee based on the information from 
detectors. Rules ma>- include such acli\ ities as question routing and archiving. 

.Solf’-as.sessnient: Method of ev’aluation vx here information specialists rate their ow n 
performance according to a set ol’ predetermined standards. 

Service roles: Personnel positions found in AskA serv’iec organi/.ations. Roles may 
inelude administrators, evpens, information specialists, team leaders, and moderators. 

Shrink-\vrai)pecl .software: b'ull}- functional and easily installed .software to help 
automate AskA ser\-iee proeesses. 

Stakeholders: Outside agencies that have a guiding interest in a digital referenee serx'iee. 

Suiiirnative evaluation: Method of evaluatit)n that focuses on the measurement ol‘ 
qualitx' often for ultimate decision making purposes. (H.vamplc: 'fo what e.xtent is the 
AskA service aehie\ing its goals?) 

1'CIVIP ( IVansniis.sion Control Pro(ocol/lriternet\vorkinj» Protocol): Software iiiles 
for allowing disparate networks to comnuimeatc. 

'rrackin<»: Process of capturing transaction information as cjuesiions arc routed througli 
an AskA serxiee. 

'I'riajie: 'Phe sorting and allocation of questions in Ask.A services iieeording to a system 
of j)riorities designed to ma,\imi/^e efficient response. 

I'nohlru.sive evaluation: Method of exaluating a serxieo xxhere the test subjects are 
Linaxx are of the lest taking place. 

X'irtiJul Reference Desk (\'RD): A project that is creating a national coo|)eralix’e digital 
referenee service in order to link the K- 1 2 community with expertise to satisfy 
information needs, \'RD seeks to identify and provide resources to organi/ations and 
individuals involxed in providing knoxx ledge and expertise to the K-12 community. 
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AskA Service Directory 



This directory includes AskA serviecs highlighted in this Starter Kit. Additional AskA 
services serving K-12 students, educators, and oihcis are available at the VRD AskA+ 
Locator liflp:/Av\nv. vnLor^/locator 

Ask Dr. Math 

hup:// forum. swanluiwrc.edii/dr.imnh/ 
dt\mcith(^ forum. s\v(irthmorc.cdn 

Ask Dr. Math is a qucstioivand-ansvvcr service for K-1 2 math teachers and students. 1'he 
service provides an archive that is searchable by grade level and topic and includes such 
features as Frequently Asked Questions (r'AQ), archives, and other resources. Ask Dr. 
Math is funded by the National Science Foundation and operates under the auspices of 
the Math Forum at Swarthmorc College. 

AskERlC 

h Up .7/u ’ u ' u askc ric ( ) r^ 
a skeri c@ askc ri c.or^ 



AskERIC is a personalized Inicmct-based AskA service providing education information 
to teachers, librarians, counselors, administrators, parents, and others throughout the 
United States and the world. It began in 1992 as a project of the ERIC Clearinghouse on 
Information & Tccimology at Syraeu.se University, doday. it encompasses the resources 
of the entire ERIC system arid beyond. (Sec ERIC in the Glossary.) 

Ask A NASA Scientist 

http : //imagine. y^sfc.)iasa.^oy/d()cs/ask_asiro/ask_a}i_astronomcrduml 
ask_asl ro @ hcasa re \ i^sfc. / nisa . o \ ' 

Ask a NASA Scientist (lormerly Ask a High Iincrgy Astronomer) is a c[uestion and 
answer service for tho.se interested in X-ray and gamma-ray astronomy. The service 
answers lOO^/i of all questions received in 2-3 days. The site also includes an archive of 
previously-answered questions and teacher resources including study guides and lesson 
plans. 

A,sk A Volcanologist 

hUp://volcaiuuindau)dak.cdn/v\ydocs/ask_a.hUi}{ 

Ino e-mail submissions] 

Ask a Volcanologist is a question-answering and referral service staffed by three 
pridessional volcanologists. The .service attempts t(^ answer all questions received within 
3 days. Fart of the larger VolcanoWorld Web site, a kcyword-scarchabic FAQ of 
commonly asked questions on volcant^'s, lesson plans for teachers, lessons and activities 
for students, and links to sources of other information about volcanoes are also available. 
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Ask Shamu 

littp://\v\v\\\sc(i\vorldj)r^/cisk_sh(n)}ii/asimh\\Juml 
shamu ((jKs'i'aw'orlcLori^ 

Ask Shamu is a question and answer service for questions about the ocean and marine 
animals. Ask Shamu answers 100''^ of all questions received, and answers arc shon and 
factual with minimal references. An SOO-number (1-800-23SHAMU) is provided for 
students to submit questions by telephone, and for lcachci\s to request curriculum 
materials. The site also features a b'AQ, curriculum guides, and more. 

Ask the Optometrist 

latp.'/Avw'wwisioncarc.com/cisk.hlm 
(I act or vision c a re, com 

Ask the Optometrist is a question-and-answer service I'or students, teachers, and parents 
that provides information about vision, eyes, and eye care. The site also includes a vision 
FAQ, links to eye care sites, and ailiclcs about vision and genera! eye care. 

Ask the Space Scientist 

http://in}a^e.f^sfcMasa,^ov/poetry/ask/askm(i^Jit}}]l 
ojen wakl @h()lcrc).^sfcMascL}^ov 

Ask the Space Scientist is a cjuestion-and-answer service operated by a NASA 
astronomer. The Space Scientist answers questions about everything having to do with 
space, including the planets, the solar system, the universe, .space travel, and more. The 
site features an archive of previously answered questions, FAQs, and links to other spacc- 
rclaled sites. Ask the Space Scientist is part of the education and public outreach effort of 
the Na\SA image satellite project. 

How I'hings Work 

lutp://!cmdciiil ,pliys.vif\^inia.cclu/Iuluccaion/reacliiny/IIovThin(^s\V()rk/li()}ne,hnnl 
lahde viry^inia.edii 

Founded and operated by a physics professor at the University (d' Virginia, How Things 
Work is a question-and-answer service for ail kinds of physics questions for all ages. The 
site also features a searchable index of previously-answered ciueslions. a recent questions 
list, and links to other “How Thing.s Work” resources. 

Internet Public Library 

lufp://arist()tie. ipl. ory/rcf/QliFJ 
ipIrefQyumich.edii 

One of several AskA services available through IPL, the Ask-A-Qiicstion service will not 
perform lengthy searches, but will suggest .some ideas and resources to help u.scrs find the 
information they seek on \'iilually any topic. The site tilso features a F'AQ of popular 
reference questions, a vimial ready reference collection catcgori/cd by subject, and tips 
for searching for information on Ihc Internet as well as in public libraries. 
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KidsConnect 

hiip://\\'\v\v.ala.or^/lCONN/AskKCJuml 
AskKC @ a IcLO 

KidsConnect is a question answering, help, and referral service for K-12 siudems on the 
Internet. The goal of llic service is to help students access and use the information 
available on the Internet effectively and efficiently. KidsConnect is a component of 
ICONnect, a technology initiative of the American Association of School Librarians 
(AASL), a division of the American Library Association. KidsConnect is offered in 
partnership with the Information Institute of Syracuse, Syracuse University, and is 
underwritten by Microsoft Corporation. 

The MAD Scientist Network 

htfp.VAvww. madsci. org 
mcidsci@madsci, wiisti edit 

The MAD Scientist Network is composed of over 500 scientists from around the world. 
The network answers questions in many areas of science and includes an online archive 
of question-answ'cr sets in addition to other resources. The MAD Scientist Network is 
operated out of Washington University School of Medicine, St. Louis; funding sources 
include the Howard Hughes Medical Institute and the Washington University School of 
Medicine Alumni Association. 

Morris County Public Library (New Jersey) 

Home Page: http://\v\\'\v, gti.net/niocolibl/MCLJuml 
Query Page: littp://w\v\v. gti.net/niocolibI/reJhox.liitnl 

Located in Whippany, New Jersey, reference librarians answer all types of questions 
from the general public and guide non-county residents to services in their area when 
appropriate. 

National Museum of American Art Reference Desk 

h ttp://nmaa- ryder. si. edn/referen cedesk/ 
jstahl@timaa.si.edu 

Information specialists at the museum answer questions regarding American art. Specific 
questions receive brief, factual responses, while users with broad queries arc directed to 
sources to aid in their research. The service is part of the National Museum of American 
Art of the Smithsonian Institution. 

ScienceNet 

htlp://\vw\v. sci encen ct. o rg. uk/i n fra/fonns/a.ska q. hi nil 
science I ine@hss. org 

ScienceNet is a UK-based question-and-answer .service that answers questions in the 
areas of science, engineering, technology, and medicine. The site includes a keyword- 
searchable database of previous questions and answers, downloadable BBC radio science 
broadcasts, interviews with scientists, articles, career information, and more. 
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'rhe following journal articles have been indexed and ahsiracicd for ihc Hducalional 
rlesoLirces Information Center (ERIC) database. 

Abels, E. G. (1996), The e-mail reference interxiew. RQ, (3), 345-58. (EJ 526 322) 

Discusses differences between e-mail reference interviews and those conducted 
using other media: presents a taxonomy of approaches to e-mail inter\'iews; and 
introduces a model e-mail interview, based on a project at the llnivcrsity of 
Maryland's College of Library and Information SerN’ices. 

Busha!iow-VVi)i)ur, L., De\'inney. G. S.. & Whitcomb, V\ (1996). Electronic mail 
reference scr\ icc: A study. RQ. 359-63, 366-71. (EJ 526 323) 

Describes a study that examined the use of electronic mail reference service in 
three library units at the Stale Laiiversity of .New York at Buffaio. Highlights 
include patron demographics, question classification, use patterns, and 
recommendations for further research. 

Internet Public Librarv: Same metaphors, new ser\ ice. (1997). American Uhrarics\ 

2cS32), 56-59. (EJ 539 658) 

A discussion wath the siaffof the Internet Public library (IPL), which began as a 
project at the LhTi\'crsily of Michigan, highlights its role in supplementing local 
public library services; classification; collection development policies; and the 
impact of legislation, copyright, and universal scr\’icc on IPL. A sidebar presents 
IPL user statistics. 

LaBountv, V. (1997). Reference desk on the Internet. Rook Report, /6(2), 19. (EJ 550 
8 ^ 5 ) 

The KidsConnect ser\'ice, with 145 \ oluntccrs from 39 states and eight countries, 
connects librarians, teachers, and students with curriculum-ic!:acd material and 
other information sources on the Internet. Questions received at the site (on the 
Web at hU}):/Anv\\'.a!a.or(^/iC()NN/in(lc'x.himl ore-mail at askkc&xila.or^) arc 
routed to \’olimtcer school librarians who respond within 48 hours. 

Lankes, R. D. (1995). AskERIC and the vimial library: Lessons for emerging digital 
libraries. Internet Research, 5( I ), 56-63. (EJ 505 459) 

E.xplores major issues in creating and maintaining Internet services for AskERIC, 
an educational digital library started by the Educational Resources Information 
Center (ERIC) system. Highlights include the importance of user input to shape 
the service, human intermediaries, AskERIC as a virtual library, future directions, 
and a copy of the AskERIC brochure. 
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McKee, M. B. (1995). A day in the life of a virtual librarian. School Lihniry Journal 
9/(4), 30-3. (EJ 501 696) 

The network information specialists at AskERlC, an Internet-based informatiem 
service for teachers, library media specialists, administrators, and others involved 
in education, select and deliver information resources to the information seeker 
within 48 hours. A sampling of questions and responses is provided in the format 
of a representative day. 

Pack,T. (1996). A guided tour of the Internet Public Library: Cyberspace's unofficial 

library offers outstanding collections of Internet resources. Database, /9(5), 5i- 
56. (EJ 532 868) 

Describes the Internet Public Library, developed ai the University of Michigan's 
School of Information and Library Studies. Site highlights include the reference 
center; the reading room, which includes materials in full-text: youth and teen 
services: professional information for librarians: links to Web search engines; a 
Multiuser Object Oriented (MOO) reference area; and a viiiual exhibit hall. 

Ryan. S. (1996). Reference service for the Internet community: A case study of the 
Internet Public Library Reference Division. Library & Information Science 
Research, JS(3), 24 1-59. (EJ 532 939) 

Examines the creation of the Internet Public Library Reference Division in the 
historical context of librarians’ efforts to integrate the use of technologies with 
reference services. Di.scussion considers ways that librarians have successfully 
incorporated new technologies and makes recommendations for use of the 
Reference Division and Internet usage for the reference process in general. 

Serving the Internet public: The Internet Public Library. (1996). Electronic Library, 

I4{2), 122-26. (EJ 526 267) 

Describes the Internet Public Library (IPL), which was developed at the School of 
Information and Library Studies at the University of Michigan to be a library for 
Inieinct users. Highlights include mission statement and goals, funding, s i'fing 
with volunteers, future possibilities, IPL services, and statement of principcs. 

Still, J., & Campbell, F. (1993). Librarian in a box: The use of electronic mail for 
reference. Refe'xoice Sendees Review, 2/(1), 15-18. (EJ 457 878) 

Discusses the use of electronic mail for reference services and investigates 
systems in health sciences libraries and in academic libraries. The use of 
electronic mail for services including inlcrlibrary loan, online search requests, 
literature searches, photocopies or other document delivery, reference questions, 
and purchase requests is described. 
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Tobiason, K. ( 1997). Taking by giving: KidsConncct and your media center. TcduuAo^y 
Comiection, 4{fi), 10-1 1. (EJ 554 221 ) 

Discusses KidsConnecl (KC), an initiative of the American Association ol‘ School 
Librarians (AASL). volunteer service to help children access and use the 
information available on the Internet effectively and efficiently. Describes the 
benefits to volunteers: improved Internet skills; knowledge of IntciTici-rclaled 
resources; familiarity with wonhwhile educational Web sites; collegial support; 
the thrill of information seeking and retrieval; and professional renewal. 
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information on pricing, shipping costs and/or other services offered by the contractor. 

Inquiries about ERIC may be addressed to the ERIC Clearinghouse on Information & 
Technology, 4-194 Center for Science and Technology, Syracuse University, Syracuse, 
NY 13244-4100 (800-464-9107), e-mail: eric@ericir.syr.edn\ or ACCESS ERIC, 2277 
Research Boulevard, 7A, Rockville, MD 20850 (800-LET-ERIC), e-mail: 
acceric@m et. ed.gov 

Journal Articles 

Copies of journal articles can be found in library periodical collections; through 
interlibrary loan; from the journal publisher; or from article reprint services such as the 
UMl/InfoStore (1-800-248-0360), UnCover Company (1-800-787-7979), or Institute for 
Scientific Information (ISI) (1-800-336-4474). Information needed for ordering includes 
the author, title of article, name of journal, volume, issue number, page numbers, date, 
and EJ number for each article. Fax services are available. 
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What is ERIC? 

ERIC, ihe Educalionai Resources Inrormalion Cenler, is a national education information 
system sponsored by the Office of Educational Research and Improvement in the L'.S. 
Department of Education. The main product of ERIC is a bibliographic database 
containing citations and abstracts for o\'cr 950.000 documents and journal articles 
published since 1966. Most of the document literature cited in ERIC can be read in full 
text at any of the 900+ librane. or institutions worldwide holding the ERIC microfiche 
collection. In addition, users can purchase copies of ERIC documents from the ERIC 
Document Reproduction Service. Journal articles cited in ERIC can be obtained at a 
subscnbing library, through intcrlibrar\' loan, or from an anicle reprint ser\‘ice 

How do 1 find information in ERIC? 

The ERIC Database can be searched manually through its two print indexes. Resources in 
Education (RIE) and Current Index to Journals in Education (CUE). 0\ er 3.000 libraiies 
and information centers subscribe to one or both of these monthly indexes. The database 
can also be searched online: (a) through a computer based information retries’al service: 
(b) by CD-ROM: (c) on a locally mounted system, which may be accessible through the 
Internet; or (d) Internet: http://ericir.syr.edu/Eric/. The number of libraries offering online 
and CD-ROM search services is rapidl\- increasing. 

What is ERIC/n ? 

The ERIC Clearinghouse on Information Technology, or ERIC/IT, is one of 16 
clearinghouses in the ERIC system. It specializes in library and information science and 
educational technology. ERIC/IT acquires, selects, catalogs, indexes, and abstracts 
documents and journal articles in these subject areas for input into the ERIC database. 
Among the topics co\crcd in library and information science are: 

• management, operation, and use of libraries and information centers 

• library technology and automation 

• library education 

• information policy 

• information literacy 

• information storage, processing and retrieval 

• networking 

Topics coN cred in educational technology include: 

• design, de\elopment, and evaluation of instruction 

• computer-assisted instruction 

• hypermedia, interacti\c \ ideo, and interactive multimedia 

• telecommunications 

• film, radio, tclc\ ision, and other audio-\ usual media 

• distance education 

• simulation and gaming 
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What is available from ERIC/IT? 

Each year, ERIC/IT publishes Monographs, Digests, and Minibibliographies in the .fields 
of educational technology and library and information science. Our semiannual 
newsletter, ERIC/IT Update, announces new clearinghouse products and developments, 
and ERIC/IT Networkers provide helpful information for using ERIC-related resources 
on the Internet. 

Publications 

• Digests, providing brief overviews of topics of cuirenl interest and references for 
further reading 

• Monographs, featuring trends and issues analyses, synthesis papers and annotated 
bibliographies 

• ERIC/IT Update, a semi-annual newsletter 
User Services 

• Response to inquiries about ERIC and matters within the ERIC/IT scope area 

• Workshops and presentations about ERIC and database searching 

• Assistance in searching the ERIC database 

AskERIC 

• Internet-based question answering service for educators 

• AskERIC Virtual Library, an Internet site of education-related information resources 
including lesson plans, InfoGuides, listservs and much more 

E-mai 1 : askeric@askenc. org 
Internet: hnp://\v\v\v,cLskenc.org 

Would you like to submit your work to ERIC? 

Have you written materials related to educational technology or library and information 
science that you would like to share with others'? ERIC/IT would be interested in 
reviewing your work for possible inclusion in the ERIC database. We actively solicit 
documents from researchers, practitioners, associations, and agencies at national, state, 
and local levels. ERIC documents include the following and more: 

• Research Reports 

• Program Descriptions 

• Instructional Materials 

• Conference Papers 

• Teaching Guides 

• Opinion Papers 
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How do 1 find out more? 

For addiiional informalion about HRIC or about submitting documents, or tor a cuircni 
publications list, contact: 

ERIC Clearinghouse on Information & Technology 
4-194 Center for Science and Technology 
Syracuse Uni\‘crsity 
Syracuse, New York 13244-4100 
R. David Lankes, Director 

Telephone: (315)443-3640 Fax: (315)443-5448 (800)464-9107 

E-mail: cric@ericir.syr.edn WWW URL: http://ericir.syr.edu/itlu)me 

Questions about the ERIC system can also be directed to: 

ACCESS ERIC 
2277 Research Poulevard, 7A 
Rockville, Maryland 20850 
Telephone: (800) LET-ERIC 

I n le m e t : ucce ric (S^ // 1 ct. ed. go \ ' 

J]Up://\v\\'\\\cispensys.c()}n/eric/ 

ERIC Clearinghouses 

• Adult, Career, and Vocational Education 

• Assessment and Evaluation 

• Community Colleges 

• Counseling and Student Services 

• Disabilities and Gifted Education 

• Educational Management 

• Elementary and Early Childhood Education 

• Higher Education 

• Informalion & Technology 

• Languages and Linguistics 

• Reading, English, and Communication 

• Rural Education and Small Schools 

• Science, Mathematics, and Environmental Education 

• Social Studies/Social Science Education 

• Teaching and Teacher Education 

• Urban Education 

Support Components 

• ERIC Document Reproduction Service 

Telephone: (800) 443-ERIC (3742) 

• ERIC Processing and Reference Facility 

Telephone: (800) 799-ERiC (3742)^ 
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Appendix A 

AskA Service Blueprints 



Each AskA service in this scclion is presented as a blucprinl. These blueprints serve as 
both examples of how plans can be structured and eases that can be emulated. The full 
method for developing these blueprints, as well as more complete, standardized 
descriptions of each scr\ icc, can be found in the companion study to this Starter Kit. 

and Maintaining, hnernet Infonnatiofi Scn'iccs: K-I2 Digital Reference 
Services (L'dnkcs. 1998). 

The Blueprint Metaphor 

The framework of detectors, rules, and effectors presented in Module 2 organizes the 
multiple ideas and data from the planning process into transferable sets of detectors, 
rules, resources, and effectors. Just as an architect's blueprint can describe a rich, 
multidimensional construction of a building in a transferable way, these blueprints 
capture the multifaceted digital reference service into a succinct, transferable description. 

However, as with architectural blueprints, there are limitations to these empirically based 
descriptions. These service blueprints identify that a feature (descriptor, rule, effector) is 
present, but not why it is present. Further, these descriptions do not capture the dynamic 
nature of the service development process in the scn.se that they do not show how rules, 
detectors, and effectors came into being or change over time. They do indicate any 
iterative functions that exist in the service as presently structured. 

The blueprints can be read left to right, with the leftmost column listing detector 
information. These detectors arc segmented veilically by agent types from the conceptual 
framework (users, inibrmation services, application builders, infrastructure providers, 
internal, and external). An agent type will not be listed if it is not used. Certain detectors 
arc marked as “Key Detectors.’' These arc detectors upon which the organization relies as 
j the most important information sources when building and maintaining its service, 

Certain detectors are connected to processes in place at the service. These arc the rules 
used to process detector information and produce effectors. Effectors are marked in black 
I with white type. Each component of the blueprint (detector, rule component, effector) can 

have one or more resources associated with it. Resources are physical mechanisms or 
I component mechanisms used at a certain point in the process. Figure 2-2 graphically 

depicts the structure of the blueprint. 

' BEST COPY AVAILABLE 
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Figure A-1 Structure of Blueprints 

The followins pages coniain blueprints of three exemplary AskA services: Ask Dr. Math, 
the Nation;*’ Museum of American Art Reference Desk, and the MAD Scientist Network. 
See Module 2 for case studies of each of these services. 

These and other blueprint figures may be seen in full size at URL: 
hiip://ericinsyr,edii/i[home/bmiis 
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Build an AskA Service in 6 Steps 











JL r^lMng^^Brepare^tmining^prQgr 

**'~ tv’.*^'.'***** '■ ' "• 

C// ' 

Protg^in|j|rao^ service 

and Go Live! 







Use the AskA Starter Kit as a guide for building and operating digital reference, 

or AskA, services! 

This set of six self-instructional modules prepares organizations and individuals to create an 
Internet-based human-mediated information service. Real-life experiences from existing 
digital reference services like AskERIC, the National Museum of American Art 
Reference Desk, NASA's Ask the Space Scientist, KidsConnect, and Ask Dr. Math 

provide helpful hints for new services. 

/ wish we had this 'early on at the National Digital Library. 
The sections on training and on evaluation 
are particularly good. 

- Martha Dexter, Library of Congress 



I learned lots from it, even after doing digital reference 
for five years. I wish 1 had this when I started. 

- Joan Stahl, National Museum of American Art, Smithsonian Institution 
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